The specific cause of the issue above has not been found but I now know how this issue can happen vs. how not to have this issue. The two final tests were not modifying the coredns configmap as agressively as outlined in https://istio.io/latest/docs/setup/install/multicluster/gateways/ and deploying k8s in a different way. 

I have been deploying KUD k8s on VMs using the KUD job, as follows:

kubectl create f 
apiVersion: batch/v1
kind: Job
metadata:
  name: kud-$CLUSTER_NAME
spec:
  template:
    spec:
      hostNetwork: true
      containers:
        - name: kud
          image: github.com/onap/multicloud-k8s:latest
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - name: multi-cluster
            mountPath: /opt/kud/multi-cluster
          - name: secret-volume
            mountPath: "/.ssh"
          command: ["/bin/sh","-c"]
          args: ["cp -r /.ssh /root/; chmod -R 600 /root/.ssh; ./installer --cluster $CLUSTER_NAME --plugins onap4k8s"]
          securityContext:
            privileged: true
      volumes:
      - name: multi-cluster
        hostPath:
          path: /opt/kud/multi-cluster
      - name: secret-volume
        secret:
          secretName: ssh-key-secret
      restartPolicy: Never
  backoffLimit: 0

If, instead, I use the same exact setup and VMs but go into the VMs and manually install KUD there using:

kud/hosting_providers/baremetal/aio.sh

Then Istio will install correctly and I am able to test network connectivity between the 2 clusters.
It is possible that some some KUD/ICN plugins are interfering with Istio because I have also disabled them when installing KUD manually via the aio.sh script. The reason I did it was the current failure of QAT plugin to install.
To confirm, I have reproduced failure on KUD job 2 extra times and success on KUD aio.sh 2 extra times.
The extra struggles around Istio certainly provided a better understanding of Istio than if no issues had been faced.
Closing the story now.

Still blocked.

Additionally, attempted to start with a fresh deployment and now am blocked by a totally new issue that wasn't an issue 1-2 weeks ago:

New pastebin of new issue: https://pastebin.com/JYwsCYjY

It seems like it's expecting a proxy but none are set (unless new code is setting them) and none are needed.

Blocked due to lacking connectivity between istio clusters:

https://istio.io/latest/docs/setup/install/multicluster/gateways/

Can’t get traffic to go from one cluster to the other:

$ kubectl exec --context=$CTX_CLUSTER1 $SLEEP_POD -n foo -c sleep – curl -I httpbin.bar.global:8000/headers

The command is from the link above. That curl fails with a connection refused.

The best I can find in the logs so far is this from the istio-proxy container of the $SLEEP_POD pod:

transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.244.0.3:53: no such host

A full log of that same istio-proxy container is here: https://pastebin.com/g1sDSxMG