Azure Self-managed Kubernetes High Availability for Open5gs [part 3]

[Part 3] Deploy Open5gs, 5G Simulation and Access to Rancher, Open5gs webgui , Kiali and Grafana

open5gs Lab Topology

source : https://assyafii.com/docs/5g-cloud-native-simulation-with-open5gs/

Only Controller-0 node

SSH to controller-0 Nodes

ssh kuberoot@20.106.131.198

Clone repo for simulation

cd ~
git clone https://github.com/indoslab/opensource-5g-core-service-mesh.git
git clone https://github.com/indoslab/openverso-charts.git

Configure AMF Services (optional)

This services if you want use NodePort & can access from external cluster.

cd ~/opensource-5g-core-service-mesh/helm-chart
cat templates/amf-deploy.yaml

Uncomment in AMF service amf-open5gs-sctp

sd is ffffff in amf-configmap

cat templates/amf-configmap.yaml

Check mongodb configuration

We will use ROOK Ceph for persistent volume, need comment existing persistent volume & add StorageClassName in persistent volume claim.

cat templates/mongodb.yaml

Deploy open5gs with helm-chart

cd ~/opensource-5g-core-service-mesh/helm-chart
helm -n open5gs install -f values.yaml open ./

Make sure all POD & Services already running

kubectl -n open5gs get pods --watch

kubectl -n open5gs get pods

kubectl -n open5gs get svc

Configure Access to Open5Gs dashboard

cat <<EOF | sudo tee open5gs-webui-gateway.yaml 
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: open5gs-webui-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: open5gs-webui
spec:
  hosts:
  - "*"
  gateways:
  - open5gs-webui-gateway
  http:
  - match:
    route:
    - destination:
        host: open-webui.open5gs.svc.cluster.local
        port:
          number: 80
EOF
kubectl apply -f open5gs-webui-gateway.yaml
NODE_PORT_OPEN5GS=$(kubectl -n istio-system get svc istio-ingressgateway \
  --output=jsonpath='{range .spec.ports[1]}{.nodePort}')
echo $NODE_PORT_OPEN5GS
31341
exit

Create firewall rule for Open5Gs dashboard

this is for simulation purpose, not recommended using this, other option available, use port from previous command 32102

az network nsg rule create -g my-resource-group \
  -n kubernetes-allow-open5gs-webui \
  --access allow \
  --destination-address-prefix '*' \
  --destination-port-range 31341\
  --direction inbound \
  --nsg-name kubernetes-nsg \
  --protocol tcp \
  --source-address-prefix '*' \
  --source-port-range '*' \
  --priority 1004

Get worker-0 external ip

az vm list-ip-addresses -g my-resource-group  -n worker-0 --query "[].virtualMachine.network.publicIpAddresses[*].ipAddress" -o tsv
20.124.98.81

Open5Gs webgui url

http://20.124.98.81:32102

Register UE (User Equipment in Open5Gs dashboard)

Open dashboard use IP webui services http://20.124.98.81:32102

Username : admin
Password : 1423

Register User Equipment (UE) with detail bellow :

IMSI : 208930000000001
Key : 465B5CE8B199B49FAA5F0A2EE238A6BC
OP : E8ED289DEBA952E4283B54E88E6183CA
opType: OPC
apn: internet
sst: 1
sd: "ffffff"

Configure UERANSIM (UE & gNB)

Install UERANSIM Helm depedency

cd ~/openverso-charts/charts/ueransim
helm dep update ./

check value in UE

sudo cat values.yaml
mcc: '208'
mnc: '93'
tac: '7'

Change AMF Address in gNB

You must change address to AMF POD address, check with below command

kubectl get pod -o wide -n open5gs | grep amf
AMF_POD_NAME=$(kubectl get pods -o=name -n open5gs | grep  open-amf | awk -F"/" '{print $2}')
AMF_ADDR=$(kubectl -n open5gs get pod $AMF_POD_NAME --template={{.status.podIP}})
echo ${AMF_ADDR}
10.244.27.208
sed -i "s/\${AMF_ADDR}/${AMF_ADDR}/g" resources/gnb.yaml

sudo cat resources/gnb.yaml

Running UERANSIM

helm -n open5gs install -f values.yaml ueransim ./
kubectl get pod -n open5gs | grep ueransim

Verify Logs UE Connected to gNB & AMF

AMF Logs

kubectl -n open5gs logs $AMF_POD_NAME

gNB Logs

kubectl -n open5gs logs ueransim-0 -c gnodeb

UE Logs

kubectl -n open5gs logs ueransim-0 -c ues

Getting access to Rancher

Getting rancher url

kubectl -n cattle-system expose deployment rancher --port=443  --name=rancher-https --type NodePort
NODE_PORT_RANCHER=$(kubectl -n cattle-system get svc rancher-https \
  --output=jsonpath='{range.spec.ports[0]}{.nodePort}')
echo $NODE_PORT_RANCHER
32536
https://20.124.98.81:32536

Create firewall rule for Rancher admin

this is for simulation purpose, not recommended using this, other option available

az network nsg rule create -g my-resource-group \
  -n kubernetes-allow-rancher-https \
  --access allow \
  --destination-address-prefix '*' \
  --destination-port-range 32536 \
  --direction inbound \
  --nsg-name kubernetes-nsg \
  --protocol tcp \
  --source-address-prefix '*' \
  --source-port-range '*' \
  --priority 1005

Getting Rancher password for admin

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'
trnf5dgqf4wqqxtqm9gqjp58dhzs2lz6z6cctqnhpqhfbxsmrkk77c

Verify ping to Internet via Rancher dashboard

UE Container

Note : Bellow not show reply, because in TAP Interface (debian) not showing reply, we can see tcpdump in UPF POD.

ping 8.8.8.8 -I uesimtun0

UPF POD

We can see ICMP request reply, from UE IP Address & Google IP.
tcpdump -i ogstun

Getting access to Grafana

Getting Grafana url

kubectl -n istio-system expose deployment grafana --port=3000  --name=grafana-http --type NodePort
NODE_PORT_GRAFANA=$(kubectl -n istio-system get svc grafana-http \
  --output=jsonpath='{range.spec.ports[0]}{.nodePort}')
echo $NODE_PORT_GRAFANA
31612
http://20.124.98.81:31612

Create firewall rule for Grafana

this is for simulation purpose, not recommended using this, other option available

az network nsg rule create -g my-resource-group \
  -n kubernetes-allow-grafana-http \
  --access allow \
  --destination-address-prefix '*' \
  --destination-port-range 31612 \
  --direction inbound \
  --nsg-name kubernetes-nsg \
  --protocol tcp \
  --source-address-prefix '*' \
  --source-port-range '*' \
  --priority 1006

Getting access to Istio Kiali Dashboard

Getting Istio Kiali Dashboard url

kubectl -n istio-system expose deployment kiali --port=20001  --name=kiali-http --type NodePort
NODE_PORT_KIALI=$(kubectl -n istio-system get svc kiali-http \
  --output=jsonpath='{range.spec.ports[0]}{.nodePort}')
echo $NODE_PORT_KIALI
31882
http://20.124.98.81:31882

Create firewall rule for Istio Kiali Dashboard

this is for simulation purpose, not recommended using this, other option available

az network nsg rule create -g my-resource-group \
  -n kubernetes-allow-kiali-http \
  --access allow \
  --destination-address-prefix '*' \
  --destination-port-range 31882 \
  --direction inbound \
  --nsg-name kubernetes-nsg \
  --protocol tcp \
  --source-address-prefix '*' \
  --source-port-range '*' \
  --priority 1007

Really appreciated the info from below references, if any question just ask :)

Enjoy!!!

References:

• https://github.com/open5gs/open5gs
• https://github.com/aligungr/UERANSIM/wiki/Installation
• https://assyafii.com/docs/5g-cloud-native-simulation-with-open5gs/
• https://medium.com/rahasak/5g-core-network-setup-with-open5gs-and-ueransim-cd0e77025fd7
• https://levelup.gitconnected.com/opensource-5g-core-with-service-mesh-bba4ded044fa
• https://github.com/Gradiant/openverso-charts.git
• https://github.com/ivanfioravanti/kubernetes-the-hard-way-on-azure