WireGuard setup Openwrt with VXLAN

Using the latest openWRT/LEDE image (openwrt-21.02.0-rc3-x86-generic-generic-ext4-combined) for WireGuard on GNS3. By default we must to change Openwrt network IP(192.168.100.1 and 192.168.100.2), GW (192.168.100.254) and add DNS (8.8.8.8):

Also change DNS settings to 8.8.8.8 :

    

For this using the following network setup:

For WireGuard we must install following packages (the same thing can be done in GUI "System" --> "Software" section).

opkg install luci-proto-wireguard wireguard-tools luci-app-wireguard kmod-wireguard

Initial setup of Wireguard is privarte and publick key generation on both nodes and setup via luci interface

mkdir /VPN
cd /VPN
#Generating public key
wg genkey > pub.key
#Generating private key
wg pubkey < pub.key  > pri.key

After generating private and public keys we must setup WireGuard via luci (WebGUI) inteface by creating new network interface  "Add new Interface..." in Network --> Interface.

The configuration on Node 192.168.100.2

First we must create WireGuard interface, the name of interface is wg0:

After it we setup new interface be defining, "Private Key", which was generated in last step, also good idea to setup Listen Port (exp 51820) and the IP of the Wiregaud interface (10.0.0.2/24) on the node OPENWRT_192.168.100.2, on OPENWRT_192.168.100.1 we will use 10.0.0.1/24

After it we must setup remote node in "Peers" add "Add peer": 

And setting for remote node the following fields:

Description : 192.168.100.2

Public Key: ENleaCdfs   - the file pub.key in other node (OPENWRT_192.168.100.1)

Allowed IPs: 10.0.0.0/24

Endpoint Host: 192.168.100.2

Endpoint Port: 51820

Persistent Keep Alive: 10 - optional settings, this will help work from behind the NAT and also tunnel will be up fro all time.

Also for testing you should put the new Wireguard interface  to lan zone, after create separate security zone

The same configuration must be applied to other node. To activate the change or after modification you need to restart network service in menu "System" --> "Startup".

To check status status of WireGuard from CLI is just ping the remote node to use the wg comand to see the status of tunnel. The bad thing that you will not see any active socket open in netstat -nap command.

The same info can be seen from GUI in "Status" --> "WireGuard" menu section.

If tunnel is down (transfer is none) or the remote node does not respond to ping or  always good to check the wireshark on GNS3 or tcpdump on node also to see in wg if the remote public key matches local peer key. Also to see if the network node you are pinging is in routing table. The bad thing that you will not see any active socket open in netstat -nap command.

VXLAN configuration

To set up the L2 connectivity from one node to other we will use vxlan, it's an encapsulation technique to encapsulate OSI layer 2 Ethernet frames in layer 4 UDP datagrams (RFC4789). 

For it we will need to install following packages:

opkg install vxlan kmod-vxlan luci-proto-vxlan

 

After installing the needed packages we will have to create net network interface "Add new interface..." in "Network" --> "Interfaces". As we are running IPv4 network we use the VXLAN (RFC7348) interface type.

Also for first/testing we setup the VXLAN interface binding on LAN interface, in this case the traffic will not be encrypted, the configuration should be as follows: 

Like in WireGuard configuration VX0 interface should be  added to Firewall-zone: lan, to allow full access for testing.

To test IP connectivity we will create new BRIDGE interface with IP. The new Bridge will connect VX0 interfaces  to LAN intefaces eth1.

In OpenWRT new bvridge interface is create in Network --> Interfaces --> Device  "Add device configuration"

And after it create IP interface in "Interfaces" section and assign IP address on 172.16.0.1 on left node and 172.16.0.2 on right node.

At this point the IP connectivity should be working after restarting network service or after reboot, the issue with vxlan as in WireGuard, the netstat does not show socket usage. So classically only Wireshark or TCP dump can show VXLAN traffic:

The configuration of the VXLAN interface vx0 in OpenWRT:

And ofcource the tcpdumo traffic sniff on vx0 intergface to see if our traffic can pass the new WireGuard and VXLAN tunnels:

In this configuration we can pass traffic from one end to other via L2 link and even have trunk connection between to switcher connected at the end of routers:

To see if traffic pass correctly we should see from sniff made on link on node "Switch1" and OPENWRT_192.168.100.1, we should see VLAN tag and traffic passing from PC1 to PC2.

The main  disadvantage (nasty thing) is that the MTU/frame size must be bellow MTU of the vx0 interface (in our case less then 1370 Bytes), also including the Ethernet and vlan headers. So setting the network we must set the PC1 and PC2 MTU size much smaller, also keeping in mind that we could be using QinQ or other encapsulation methods.

A topic for L2 fragmentation i think:)

Python Elasticsearch

Basic ElasticSearch connection to from Python and search, the main issue is to convert the retrieved data to your suitable  needs.  For begging limit the data we receive by using size option in (size=10), and limit the retrieving fields, only the needed one ("_source": ["field_x", ..., "field_y"],).

the biggest issue is the the nested dict we receive from  ElasticSearch, to convert to DataFrame we musrt to use the json_normalize.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58

# Elasticsearch stuff to import import ssl, certifi from elasticsearch import Elasticsearch from elasticsearch.connection import create_ssl_context from elasticsearch import Elasticsearch, RequestsHttpConnection # Panda stuff to import from pandas import json_normalize def main_search1(): # # no certificate verification ssl_context = create_ssl_context() ssl_context.check_hostname = False ssl_context.verify_mode = ssl.CERT_NONE es = Elasticsearch(hosts=[{'host': '127.0.0.1', 'port': 9200}], scheme="https", # to ensure that it does not use the default value `True` connection_class=RequestsHttpConnection, # enable SSL use_ssl=True, verify_certs=False, http_auth=("user", "password")) print (es.info()) # search query on elasticsearch result = es.search( index="syslog-2021.03.12", body={ # field to retriev from elasticsearch "_source": ["cisco", "timestamp"], # search query "query": { "match": { 'user.name':'test' } } }, # number of results to retriev size=10) # show retriewed result "['hits']['hits']" show only found data print(result['hits']['hits']) # print results from Elasticsearch all_hits = result['hits']['hits'] for num, doc in enumerate(all_hits): print ("DOC ID:", doc["_id"], "--->", doc, type(doc), "\n") #convert to Panda DataFrame with normalize dictiniory --> https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html res_content_pd = json_normalize(result['hits']['hits']) print (res_content_pd) return if __name__ == '__main__': main_search1()

Elasticsearch-dsl

The main problem of using Elasticsearch API is the query (body) syntax, it's not human friendly especial for first time or mass usage in code, it's hard to write debug and execute correctly.

The main idea of Elasticsearch-dsl is to simplify the query and filters of API.

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

# Elasticsearch stuff to import import ssl, certifi from elasticsearch import Elasticsearch from elasticsearch.connection import create_ssl_context from elasticsearch import Elasticsearch, RequestsHttpConnection # Panda stuff to import from pandas import json_normalize from elasticsearch_dsl import Search, Q def main_search2(): # # no certificate verification ssl_context = create_ssl_context() ssl_context.check_hostname = False ssl_context.verify_mode = ssl.CERT_NONE es = Elasticsearch(hosts=[{'host': '127.0.0.1', 'port': 9200}], scheme="https", # to ensure that it does not use the default value `True` connection_class=RequestsHttpConnection, # enable SSL use_ssl=True, verify_certs=False, http_auth=("user", "password")) print (es.info()) # search query on elasticsearch-dsl, more simple way ti make logical queries # if searchig for nested data (example user:{name:'test1'} we must use double ** if not nested no ** needed query = Q('match', **{'user.name':'test'}) & Q('match', **{'observer.ip':'1.1.1.1'}) #difine index ant result numbet to retriev with size option s = Search(using=es, index='syslog-2021.03.12').query(query).extra(size=4000) # define fields to retriev fields(['timestamp', 'cisco']) s = s.source(['timestamp', 'cisco']) #count the number of resuls total = s.count() print(total) #difine tthe numbet of results to retriev s = s[0:10] # Execute function search and return an instance of Response wrapping all the data. # if retrieving big data set use scan() option which returns a generator that will iterate over all the documents matching the query. res_content = s.execute() # show retriewed result "['hits']['hits']" show only found data print(res_content['hits']['hits']) # print results from Elasticsearch all_hits = res_content['hits']['hits'] for num, doc in enumerate(all_hits): print ("DOC ID:", doc["_id"], "--->", doc, type(doc), "\n") # convert to Panda DataFrame with normalize dictiniory --> https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html results= [d.to_dict() for d in res_content] res_content_pd1 = json_normalize(results) print(res_content_pd1) # not os effiecient way of using ['hits']['hits'], not make dataframe (more time is needed ) res_filtered = [x['_source'].to_dict() for x in res_content['hits']['hits']] res_content_pd2 = json_normalize(res_filtered) print (res_content_pd2) if __name__ == '__main__': main_search2()

The main difference of the Elasticsearch and Elasticsearch-dsl is the query fields:

query = Q('match', **{'user.name':'test'}) & Q('match', **{'observer.ip':'1.1.1.1'})

We can define the match query and logical values or and, etc..

To match field within another field we must use ** for defining nested dict sub values by dot.

More info here --> https://elasticsearch-dsl.readthedocs.io/_/downloads/en/6.3.0/pdf/

Section "Dotted fields"