- Initiate the Local Broker wizard.
sudo ./npa_localbroker_wizard
- The Wizard provides an overview of Local Broker status.
- The Wizard also allows admins to upgrade, collect troubleshooting files, and set log levels.
- Check Network Events to review traffic details.
- Check if the services are running.
sudo microk8s kubectl get svc -n npa-dp-localbroker
- Check if the pods are running.
sudo microk8s kubectl get pods -n npa-dp-localbroker
- Verify if registered successfully message is seen during registration.
- Check the version of the Local Broker.
Command Line:sudo microk8s kubectl -n npa-dp-localbroker describe pods | grep -i image Image: netskopeprivateaccess/npa-stproxy-lbr:116.0.0-18047 Image ID: docker.io/netskopeprivateaccess/npa-stproxy-lbr@sha256:4b49d6a9b86e08167e5380d130f722161e5e87c97b2341ff5769ef36558b9b2c Image: netskopeprivateaccess/npagw-lbr:116.0.0-18047 Image ID: docker.io/netskopeprivateaccess/npagw-lbr@sha256:d9c1ba92f1b51292df77af945f94fc4fe1e4a2a6c9bda696bcbc64605845ee4f Image: netskopeprivateaccess/npastitcher-lbr:116.0.0-18047 Image ID: docker.io/netskopeprivateaccess/npagw-lbr@sha256:d9c1ba92f1b51292df77af945f94fc4fe1e4a2a6c9bda696bcbc64605845ee4f ~$UI:
Client is not connected to Local Broker NPA Gateway
- Ensure client side resolution to Local Broker hostname works.
- Ensure tcp 443 is allowed to Local Broker.
- Ensure there is no SSL decryption of traffic to Local Broker.
- Validate connectivity logs within npadebuglog.log
- Check by running outer pcap on the Client, and running tcpdump on Local Broker for the Client IP.
Publisher is not connected to the Local Broker NPA Publisher Gateway (Stitcher)
- Ensure the Publisher-side resolution to the Local Broker hostname works.
- Ensure TCP 1443 is allowed to the Local Broker.
- Ensure there is no SSL decryption of traffic to the Local Broker.
- Validate connectivity logs within the agent.txt logs
- Check by running tcpdump on the Publisher, and also tcpdump on the Local Broker for the Publisher’s IP.
Check Utilization of the Local Broker
Some useful commands such as top, htop, free -m, df -kh would be helpful to assess the health of the Local Broker.
Check logs on the Local Broker.
<hostname>:~$ sudo microk8s kubectl get pods -n npa-dp-localbroker NAME READY STATUS RESTARTS AGE local-npaproxy-7b9c4956f6-nkxrl 1/1 Running 0 3d21h local-npagw-c787785bf-zrxfd 1/1 Running 0 3d21h local-npastitcher-78dc8869db-p47mg 1/1 Running 0 3d21h <hostname>:~$
To access NPA Gateway (Client Gateway) and Stitcher (Publisher Gateway) Pods: (review server.txt and stitcher.txt logs)
Option 1:
cd /opt/ns/local/logs/ ls
Option 2: (log in to NPA Gateway/Stitcher pods)
#NPA Gateway: sudo microk8s kubectl exec -it local-npagw-c787785bf-zrxfd -n npa-dp-localbroker -- /bin/bash cd logs ls #NPA Stitcher sudo microk8s kubectl exec -it local-npastitcher-78dc8869db-p47mg -n npa-dp-localbroker -- /bin/bash cd logs ls
Restart Pods
To restart npagw:
sudo microk8s kubectl -n npa-dp-localbroker rollout restart deployment local-npagw
To restart npastitcher:
sudo microk8s kubectl -n npa-dp-localbroker rollout restart deployment local-npastitcher
To restart npastproxy:
sudo microk8s kubectl -n npa-dp-localbroker rollout restart deployment local-npa-stproxy
To restart all NPA services:
sudo microk8s kubectl rollout restart deployment -n npa-dp-localbroker
How to run Troubleshooter to Identify Issues (especially with certificates)
Use the troubleshooter option within the Wizard. Here’s an example:
Local Broker information:
Version: 123.0.0-19497
Registration information:
Netskope address: <tenant_URL>
Certificate CN: <CN>
Network settings:
Local IP address: 10.136.10.45
Gateway: 10.136.8.1
DNS: 10.136.9.106, 8.8.8.8
Default search domain: search.acme.net
Update status:
*** System restart required ***
System updates available.
Local Broker software upgrade available. Please upgrade.
Configuration menu:
1. Upgrade
2. Troubleshooter
3. Log settings
4. Exit
2
Configuration menu:
1. Generate log bundle to store related logs from syslog and local broker
2. Run troubleshooter test to check if related services / config is running properly
3. Collect metrics in the local broker service
4. Describe Local Broker Pod information
5. Return to previous menu
2
Tenant status:
Tenant is reachable.
Netskope Cloud certificate status:
Netskope Cloud certificate is valid.
AppArmor status:
apparmor.service - Load AppArmor profiles
Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2024-10-17 06:22:32 UTC; 3 months 12 days ago
Docs: man:apparmor(7)
https://gitlab.com/apparmor/apparmor/wikis/home/
Main PID: 763 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 19086)
Memory: 0B
CGroup: /system.slice/apparmor.service
Jan 14 21:11:08 npa-local-broker systemd[1]: Reloading Load AppArmor profiles.
Jan 14 21:11:08 npa-local-broker apparmor.systemd[811346]: Restarting AppArmor
Jan 14 21:11:08 npa-local-broker apparmor.systemd[811346]: Reloading AppArmor profiles
Jan 14 21:11:08 npa-local-broker apparmor.systemd[811369]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Jan 14 21:11:08 npa-local-broker systemd[1]: Reloaded Load AppArmor profiles.
Jan 25 03:21:06 npa-local-broker systemd[1]: Reloading Load AppArmor profiles.
Jan 25 03:21:06 npa-local-broker apparmor.systemd[1657602]: Restarting AppArmor
Jan 25 03:21:06 npa-local-broker apparmor.systemd[1657602]: Reloading AppArmor profiles
Jan 25 03:21:06 npa-local-broker apparmor.systemd[1657612]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Jan 25 03:21:06 npa-local-broker systemd[1]: Reloaded Load AppArmor profiles.
AppArmor policy:
apparmor module is loaded.
104 profiles are loaded.
38 profiles are in enforce mode.
/snap/snapd/21759/usr/lib/snapd/snap-confine
/snap/snapd/21759/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/snap/snapd/23258/usr/lib/snapd/snap-confine
/snap/snapd/23258/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/snap/snapd/23545/usr/lib/snapd/snap-confine
/snap/snapd/23545/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/bin/man
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/lib/snapd/snap-confine
/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/sbin/tcpdump
/{,usr/}sbin/dhclient
cri-containerd.apparmor.d
docker-default
lsb_release
man_filter
man_groff
nvidia_modprobe
nvidia_modprobe//kmod
snap-update-ns.helm
snap-update-ns.lxd
snap-update-ns.microk8s
snap-update-ns.trivy
snap.lxd.activate
snap.lxd.benchmark
snap.lxd.buginfo
snap.lxd.check-kernel
snap.lxd.daemon
snap.lxd.hook.configure
snap.lxd.hook.install
snap.lxd.hook.remove
snap.lxd.lxc
snap.lxd.lxc-to-lxd
snap.lxd.lxd
snap.lxd.migrate
snap.trivy.trivy
66 profiles are in complain mode.
snap.helm.helm
snap.microk8s.add-node
snap.microk8s.addons
snap.microk8s.config
snap.microk8s.ctr
snap.microk8s.daemon-apiserver-kicker
snap.microk8s.daemon-apiserver-proxy
snap.microk8s.daemon-cluster-agent
snap.microk8s.daemon-containerd
snap.microk8s.daemon-etcd
snap.microk8s.daemon-flanneld
snap.microk8s.daemon-k8s-dqlite
snap.microk8s.daemon-kubelite
snap.microk8s.dashboard-proxy
snap.microk8s.dbctl
snap.microk8s.disable
snap.microk8s.enable
snap.microk8s.helm
snap.microk8s.helm3
snap.microk8s.hook.configure
snap.microk8s.hook.connect-plug-account-control
snap.microk8s.hook.connect-plug-cifs-mount
snap.microk8s.hook.connect-plug-docker-privileged
snap.microk8s.hook.connect-plug-docker-support
snap.microk8s.hook.connect-plug-dot-config-helm
snap.microk8s.hook.connect-plug-dot-kube
snap.microk8s.hook.connect-plug-firewall-control
snap.microk8s.hook.connect-plug-fuse-support
snap.microk8s.hook.connect-plug-hardware-observe
snap.microk8s.hook.connect-plug-home
snap.microk8s.hook.connect-plug-home-read-all
snap.microk8s.hook.connect-plug-k8s-journald
snap.microk8s.hook.connect-plug-k8s-kubelet
snap.microk8s.hook.connect-plug-k8s-kubeproxy
snap.microk8s.hook.connect-plug-kernel-crypto-api
snap.microk8s.hook.connect-plug-kernel-module-observe
snap.microk8s.hook.connect-plug-kubernetes-support
snap.microk8s.hook.connect-plug-log-observe
snap.microk8s.hook.connect-plug-login-session-observe
snap.microk8s.hook.connect-plug-mount-observe
snap.microk8s.hook.connect-plug-network
snap.microk8s.hook.connect-plug-network-bind
snap.microk8s.hook.connect-plug-network-control
snap.microk8s.hook.connect-plug-network-observe
snap.microk8s.hook.connect-plug-opengl
snap.microk8s.hook.connect-plug-process-control
snap.microk8s.hook.connect-plug-system-observe
snap.microk8s.hook.disconnect-plug-network-control
snap.microk8s.hook.install
snap.microk8s.hook.post-refresh
snap.microk8s.hook.remove
snap.microk8s.images
snap.microk8s.inspect
snap.microk8s.istioctl
snap.microk8s.join
snap.microk8s.kubectl
snap.microk8s.leave
snap.microk8s.linkerd
snap.microk8s.microk8s
snap.microk8s.refresh-certs
snap.microk8s.remove-node
snap.microk8s.reset
snap.microk8s.start
snap.microk8s.status
snap.microk8s.stop
snap.microk8s.version
17 processes have profiles defined.
11 processes are in enforce mode.
/usr/bin/bash (452326) cri-containerd.apparmor.d
/usr/bin/python3.8 (452339) cri-containerd.apparmor.d
/root/go/bin/squid-exporter (452340) cri-containerd.apparmor.d
/usr/sbin/squid (452341) cri-containerd.apparmor.d
/usr/bin/bash (452396) cri-containerd.apparmor.d
/usr/bin/bash (452403) cri-containerd.apparmor.d
/home/newedge_gateway (452423) cri-containerd.apparmor.d
/home/newedge_gateway (452424) cri-containerd.apparmor.d
/coredns (813328) cri-containerd.apparmor.d
/usr/bin/kube-controllers (813486) cri-containerd.apparmor.d
/coredns (1980530) cri-containerd.apparmor.d
6 processes are in complain mode.
/usr/bin/bash (1657402) snap.microk8s.daemon-apiserver-kicker
/snap/microk8s/7590/bin/sleep (2485892) snap.microk8s.daemon-apiserver-kicker
/usr/bin/bash (1657394) snap.microk8s.daemon-cluster-agent
/snap/microk8s/7590/bin/cluster-agent (1657560) snap.microk8s.daemon-cluster-agent
/snap/microk8s/7590/bin/k8s-dqlite (1657682) snap.microk8s.daemon-k8s-dqlite
/snap/microk8s/7590/kubelite (1657696) snap.microk8s.daemon-kubelite
0 processes are unconfined but have a profile defined.
ufw status:
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
53/tcp on tun0 ALLOW IN Anywhere
53/udp on tun0 ALLOW IN Anywhere
22/tcp ALLOW IN Anywhere
443/tcp ALLOW IN Anywhere
Anywhere on lo ALLOW IN Anywhere
Anywhere DENY IN 127.0.0.0/8
Anywhere on vxlan.calico ALLOW IN Anywhere
Anywhere on cali+ ALLOW IN Anywhere
53/tcp (v6) on tun0 ALLOW IN Anywhere (v6)
53/udp (v6) on tun0 ALLOW IN Anywhere (v6)
22/tcp (v6) ALLOW IN Anywhere (v6)
443/tcp (v6) ALLOW IN Anywhere (v6)
Anywhere (v6) on lo ALLOW IN Anywhere (v6)
Anywhere (v6) DENY IN ::1
Anywhere (v6) on vxlan.calico ALLOW IN Anywhere (v6)
Anywhere (v6) on cali+ ALLOW IN Anywhere (v6)
Anywhere ALLOW OUT Anywhere on vxlan.calico
Anywhere ALLOW OUT Anywhere on cali+
Anywhere (v6) ALLOW OUT Anywhere (v6) on vxlan.calico
Anywhere (v6) ALLOW OUT Anywhere (v6) on cali+
Port forwarding status:
Chain PREROUTING (policy ACCEPT 1430 packets, 88682 bytes)
pkts bytes target prot opt in out source destination
1768K 171M cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */
1784K 173M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 703 packets, 43272 bytes)
pkts bytes target prot opt in out source destination
8995K 541M cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
9449K 568M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain POSTROUTING (policy ACCEPT 2062 packets, 160K bytes)
pkts bytes target prot opt in out source destination
9887K 612M cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */
9757K 596M KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
Chain KUBE-EXT-IEMTRDY6YQRMCAW4 (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* masquerade traffic for npa-dp-localbroker/npa-stproxy-service external destinations */
0 0 KUBE-SVC-IEMTRDY6YQRMCAW4 all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-EXT-STGGDS6DI3V6ZCLP (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* masquerade traffic for npa-dp-localbroker/npagw-service external destinations */
0 0 KUBE-SVC-STGGDS6DI3V6ZCLP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-EXT-XJT5JLYP4GYYYJYA (2 references)
pkts bytes target prot opt in out source destination
6 360 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* masquerade traffic for npa-dp-localbroker/npastitcher-service external destinations */
6 360 KUBE-SVC-XJT5JLYP4GYYYJYA all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-KUBELET-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-MARK-MASQ (17 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-EXT-STGGDS6DI3V6ZCLP tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npagw-service */ tcp dpt:30443
0 0 KUBE-EXT-XJT5JLYP4GYYYJYA tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npastitcher-service */ tcp dpt:30444
0 0 KUBE-EXT-IEMTRDY6YQRMCAW4 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npa-stproxy-service */ tcp dpt:30128
Chain KUBE-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
2062 160K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ random-fully
Chain KUBE-PROXY-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-SEP-4SXU5PHGPZ4M5SVW (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.1.229.22 0.0.0.0/0 /* npa-dp-localbroker/npagw-service */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npagw-service */ tcp to:10.1.229.22:4443
Chain KUBE-SEP-KNG5QBTAOEVCPHRC (1 references)
pkts bytes target prot opt in out source destination
33 1980 KUBE-MARK-MASQ all -- * * 10.136.10.45 0.0.0.0/0 /* default/kubernetes:https */
33 1980 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:10.136.10.45:16443
Chain KUBE-SEP-MJIFVF3IKKXRIIYJ (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.1.229.26 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.1.229.26:9153
Chain KUBE-SEP-TWIOLUZH4SSQBIPI (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.1.229.24 0.0.0.0/0 /* npa-dp-localbroker/npastitcher-service */
6 360 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npastitcher-service */ tcp to:10.1.229.24:4444
Chain KUBE-SEP-VWX5IKJN4SL6ZPU5 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.1.229.23 0.0.0.0/0 /* npa-dp-localbroker/npa-stproxy-service */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npa-stproxy-service */ tcp to:10.1.229.23:3128
Chain KUBE-SEP-WHSOA3L5OXOZ4QGW (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.1.229.26 0.0.0.0/0 /* kube-system/kube-dns:dns */
129K 11M DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.1.229.26:53
Chain KUBE-SEP-X4NOFRWOEY2DEEXE (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.1.229.26 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.1.229.26:53
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-STGGDS6DI3V6ZCLP tcp -- * * 0.0.0.0/0 10.152.183.151 /* npa-dp-localbroker/npagw-service cluster IP */ tcp dpt:443
0 0 KUBE-EXT-STGGDS6DI3V6ZCLP tcp -- * * 0.0.0.0/0 10.136.10.45 /* npa-dp-localbroker/npagw-service external IP */ tcp dpt:443
0 0 KUBE-SVC-XJT5JLYP4GYYYJYA tcp -- * * 0.0.0.0/0 10.152.183.45 /* npa-dp-localbroker/npastitcher-service cluster IP */ tcp dpt:1443
0 0 KUBE-EXT-XJT5JLYP4GYYYJYA tcp -- * * 0.0.0.0/0 10.136.10.45 /* npa-dp-localbroker/npastitcher-service external IP */ tcp dpt:1443
0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.152.183.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.152.183.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SVC-JD5MR3NA4I4DYORP tcp -- * * 0.0.0.0/0 10.152.183.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
1359 117K KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.152.183.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
0 0 KUBE-SVC-IEMTRDY6YQRMCAW4 tcp -- * * 0.0.0.0/0 10.152.183.53 /* npa-dp-localbroker/npa-stproxy-service cluster IP */ tcp dpt:5000
0 0 KUBE-EXT-IEMTRDY6YQRMCAW4 tcp -- * * 0.0.0.0/0 10.136.10.45 /* npa-dp-localbroker/npa-stproxy-service external IP */ tcp dpt:5000
274 16977 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SEP-X4NOFRWOEY2DEEXE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp -> 10.1.229.26:53 */
Chain KUBE-SVC-IEMTRDY6YQRMCAW4 (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.53 /* npa-dp-localbroker/npa-stproxy-service cluster IP */ tcp dpt:5000
0 0 KUBE-SEP-VWX5IKJN4SL6ZPU5 all -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npa-stproxy-service -> 10.1.229.23:3128 */
Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
0 0 KUBE-SEP-MJIFVF3IKKXRIIYJ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics -> 10.1.229.26:9153 */
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
pkts bytes target prot opt in out source destination
33 1980 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
33 1980 KUBE-SEP-KNG5QBTAOEVCPHRC all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https -> 10.136.10.45:16443 */
Chain KUBE-SVC-STGGDS6DI3V6ZCLP (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.151 /* npa-dp-localbroker/npagw-service cluster IP */ tcp dpt:443
0 0 KUBE-SEP-4SXU5PHGPZ4M5SVW all -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npagw-service -> 10.1.229.22:4443 */
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ udp -- * * !10.1.0.0/16 10.152.183.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
129K 11M KUBE-SEP-WHSOA3L5OXOZ4QGW all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns -> 10.1.229.26:53 */
Chain KUBE-SVC-XJT5JLYP4GYYYJYA (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.45 /* npa-dp-localbroker/npastitcher-service cluster IP */ tcp dpt:1443
6 360 KUBE-SEP-TWIOLUZH4SSQBIPI all -- * * 0.0.0.0/0 0.0.0.0/0 /* npa-dp-localbroker/npastitcher-service -> 10.1.229.24:4444 */
Chain cali-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
8995K 541M cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:GBTAv2p5CwevEyJm */
Chain cali-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
9887K 612M cali-fip-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Z-c7XtVd2Bq7s_hA */
9887K 612M cali-nat-outgoing all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nYKhEzDlr11Jccal */
0 0 MASQUERADE all -- * vxlan.calico 0.0.0.0/0 0.0.0.0/0 /* cali:e9dnSgSVNmIcpVhP */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL random-fully
Chain cali-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
1768K 171M cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:r6XmIziWUJsdOK6Z */
Chain cali-fip-dnat (2 references)
pkts bytes target prot opt in out source destination
Chain cali-fip-snat (1 references)
pkts bytes target prot opt in out source destination
Chain cali-nat-outgoing (1 references)
pkts bytes target prot opt in out source destination
584K 43M MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully
Local Broker port mapping status is OK
lscpu :
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 40 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7662 64-Core Processor
Stepping: 0
CPU MHz: 1999.999
BogoMIPS: 3999.99
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 4 MiB
L3 cache: 2 GiB
NUMA node0 CPU(s): 0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ssbd ibpb vmmcall fsgsbase bmi1 smep arat overflow_recov succor
df -h:
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 1.7M 1.6G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 39G 21G 16G 57% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/loop0 56M 56M 0 100% /snap/core18/2829
/dev/loop3 64M 64M 0 100% /snap/core20/2379
/dev/loop1 56M 56M 0 100% /snap/core18/2846
/dev/loop7 67M 67M 0 100% /snap/core24/609
/dev/loop8 92M 92M 0 100% /snap/lxd/24061
/dev/sda2 974M 211M 696M 24% /boot
/dev/loop13 92M 92M 0 100% /snap/lxd/29619
tmpfs 170M 12K 170M 1% /var/snap/microk8s/common/var/lib/kubelet/pods/e6bd8981-6869-49c4-b3e5-e9659f5edb4b/volumes/kubernetes.io~projected/kube-api-access-z6hgw
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/150bd3ec-ae30-4769-a7ec-97e48c824c6a/volumes/kubernetes.io~projected/kube-api-access-7fhpb
/dev/loop12 64M 64M 0 100% /snap/core20/2434
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/1b49bd01d846458a20ec9586ade1513a353fe5eaad29bd2ddda6b02f067404f2/shm
shm 64M 12K 64M 1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/c5f1aef443a0863f053918259aa90845756f24cfab4762495e926f0d48272673/shm
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/4a3d8a981aefe287c2d44d96f008b9f87693a4d42dfa56a5fcb3241ad73f3fe3/shm
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/4a3d8a981aefe287c2d44d96f008b9f87693a4d42dfa56a5fcb3241ad73f3fe3/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/c5f1aef443a0863f053918259aa90845756f24cfab4762495e926f0d48272673/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/1b49bd01d846458a20ec9586ade1513a353fe5eaad29bd2ddda6b02f067404f2/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/44f9dced28ed9f2f78bf4591df4ef07d903a496a81e12695d61d877cf7489183/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/466ebee2d64cf834d878323aa46c21078404a54ce0b9885fafeb0def4452a461/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/932985beb8cc97fece53debf386ac171c371c61540203fb232215cb31d0a6e55/rootfs
/dev/loop9 45M 45M 0 100% /snap/snapd/23258
/dev/loop17 74M 74M 0 100% /snap/core22/1722
/dev/loop5 42M 42M 0 100% /snap/trivy/276
/dev/loop18 19M 19M 0 100% /snap/helm/436
/dev/loop14 45M 45M 0 100% /snap/snapd/23545
/dev/loop10 162M 162M 0 100% /snap/microk8s/7503
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/1f1a036c66089d2429f387411dd54976a88b19f647e1573baecb7f1e7410b511/shm
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/1f1a036c66089d2429f387411dd54976a88b19f647e1573baecb7f1e7410b511/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/1308a2452e98f455e61c654a842fa07892e7bb4c4ab294b3f1263b22327b938e/rootfs
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/f4a880f58501fcb4adb7eb2b18352861b6497388cd10201103c92a5b0651fd38/shm
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/f4a880f58501fcb4adb7eb2b18352861b6497388cd10201103c92a5b0651fd38/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/bf9bbe2e8230cba0a2544db48bdfc70bedd7b901ffa56e78c96f56078c809a78/rootfs
/dev/loop19 19M 19M 0 100% /snap/helm/440
/dev/loop15 67M 67M 0 100% /snap/core24/716
/dev/loop6 162M 162M 0 100% /snap/microk8s/7590
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/d1028bbd-9236-48f0-bf6c-23d29df9e0b8/volumes/kubernetes.io~projected/kube-api-access-2wzbm
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/6332a72a28ab2fc38dce004c2cf2183760d09805676f509cd996e6c2dfcc23da/shm
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/6332a72a28ab2fc38dce004c2cf2183760d09805676f509cd996e6c2dfcc23da/rootfs
overlay 39G 21G 16G 57% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/50174dacbf8b540a1781bccee773eea1d86bec887d42090f81f45e2c9ff2e96b/rootfs
/dev/loop16 74M 74M 0 100% /snap/core22/1748
tmpfs 1.6G 0 1.6G 0% /run/user/1000
cat /proc/meminfo:
MemTotal: 16380372 kB
MemFree: 1299568 kB
MemAvailable: 14675460 kB
Buffers: 858348 kB
Cached: 11109024 kB
SwapCached: 0 kB
Active: 7162440 kB
Inactive: 5957048 kB
Active(anon): 1021812 kB
Inactive(anon): 232 kB
Active(file): 6140628 kB
Inactive(file): 5956816 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 640 kB
Writeback: 0 kB
AnonPages: 1152016 kB
Mapped: 584564 kB
Shmem: 1804 kB
KReclaimable: 1616524 kB
Slab: 1854400 kB
SReclaimable: 1616524 kB
SUnreclaim: 237876 kB
KernelStack: 9932 kB
PageTables: 9540 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8190184 kB
Committed_AS: 3780760 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 44092 kB
VmallocChunk: 0 kB
Percpu: 14464 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 587648 kB
DirectMap2M: 13043712 kB
DirectMap1G: 5242880 kB
Local Broker information:
Version: 123.0.0-19497
Registration information:
Netskope address: <tenant_URL>
Certificate CN: <CN>
Network settings:
Local IP address: 10.136.10.45
Gateway: 10.136.8.1
DNS: 10.136.9.106, 8.8.8.8
Default search domain: search.acme.net
Update status:
*** System restart required ***
System updates available.
Local Broker software upgrade available. Please upgrade.
Configuration menu:
1. Upgrade
2. Troubleshooter
3. Log settings
4. Exit
Unable to upgrade Local Broker due to Docker limit
- Identifying the error: An error may occur when pulling the local broker Docker image due to Docker Hub’s rate limits.
Error message example:==> downloading helm chartError: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/netskopeprivateaccess/npa-local-broker/manifests/sha256:8a94a50e9a5be0845bf6dbdc7512a45efb6085f04f47332ba0e4c2f02989aa00: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit - Understanding rate limits: Docker Hub limits anonymous image pulls per IP address. Exceeding this limit triggers a Too Many Requests error.
- Bypassing the rate limit: Authenticate with a Docker Hub account before running the local broker installation script to avoid this issue.
- Authentication steps
- Switch to root using
sudo su. - Run docker login and enter your credentials.
- Docker stores credentials in
/home/root/.docker/config.json.
- Switch to root using
- Upgrading the LBR: When upgrading the LBR with
sudovia script or wizard, Docker will access credentials from config.json.