Develop

.gitignore

@@ -134,6 +134,7 @@ data/es_backups/
 *.retry
 .vscode
 .vimrc
+.pensando-elk
 
 
 # # Sphinx doc supporting files and dirs

README.md

@@ -1,11 +1,11 @@
-<img src="https://www.amd.com/system/files/styles/992px/private/2022-05/1423875-pensando-logo-white-1260x709_0.png?itok=D5gjoCQP" alt="AMD Pensando" width="350"/>
+<img src="https://th.bing.com/th/id/OIP.CwPiU5tKuxQpL4ZMRSoVIQAAAA?pid=ImgDet&rs=1" alt="AMD Pensando" width="350"/>
 
 
 ELK based analytics for Pensado Systems
 
 This repository is the starting point for building and utlizing the Elasticstack for monitoring and analyzing
-data, both about and traversing, the Pensando DSS(es) - DSC(s) coming soon - within your environment.  The purpose is to consolidate the
-applications and tools used for said monitoring and analysis and deploy them in an automated fashion.
+data, both about and traversing, the Pensando DSS(es) within your environment.  The purpose is to consolidate the
+applications and tools used for said monitoring and analysis and deploy them in an easy fashion.
 
 Instantiation can be done on any system with docker and docker-compose installed.
 
@@ -14,34 +14,46 @@ Instantiation can be done on any system with docker and docker-compose installed
 
 :warning: <span style="color:yellow">**WARNING**</span> :warning:
 
-<mark>DO NOT RUN DOCKER AS ROOT!!!  IT WILL NOT WORK. </mark>
-<mark>DO NOT DO *ANYTHING* AS ROOT.  NEVER SUDO TO INSTALL OR CONFIGURE THIS, EVERYTHING BREAKS!!! </mark>
+<mark>DO NOT RUN THE INSTALL OR CONFIGURATION AS ROOT!!!  IT WILL NOT WORK. </mark> <br/>
+To run docker as a non-root user, simply add the user to the docker group, then log out and log back in.
+
+```
+sudo usermod -aG docker ${USER}
+```
+
+
 
 
 <br/>
 
 ---
 **NOTE**
 
-This branch works with the following software.
+This branch works with the following software. <br/>
 
-CXOS: 10.12.x <br/>
-PSM:  1.59.0-50 or later
+CXOS: 10.15.x <br/>
+PSM:  1.100.2-T-8 or later
 
-If these do not match your current install, [check one of the other branches](https://gitlab.com/pensando/tbd/siem/elastic/elk-pensando/-/branches)
+If these do not match your current install, [check one of the other branches](https://github.com/amd/pensando-elk/branches)
 
 ---
-
+  ### Please fully read the Support Policy below if you are having problems installing or configuring this
 
   #### Installation and running
-
-  1. Clone this repository
 
-  2. Change into the directory where it is stored
+  1. Verify you are on the correct branch before starting
+        ```
+        git branch
+        ```
+
+  2. If you are not, use the following command to switch to the correct branch:
+        ```
+        git checkout develop
+        ```
 
   3. run the following command (change 8.6.2 if the version of ELK you want is different):
       ```
-      echo "TAG=8.6.2" >.env
+      echo "TAG=8.16.1" >.env
       ```
 
   4. Create the following directories and give them full write permissions (777 works)
@@ -63,19 +75,35 @@ If these do not match your current install, [check one of the other branches](ht
 
         Change false to true
         ``` bash
-            EF_OUTPUT_ELASTICSEARCH_ENABLE: 'false'
+            EF_OUTPUT_ELASTICSEARCH_ENABLE: 'true'
         ```
-            
+
         Change the "CHANGEME" in this line to the IP address of your system.  Do not use localhost or the loopback, it will not work
         ``` bash
             EF_OUTPUT_ELASTICSEARCH_ADDRESSES: 'CHANGEME:9200'
         ```
+        Enable daily log file rotation by uncommenting the line
+        ``` bash
+           EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: 'daily'
+        ```
+      Or:
+      ```
+      localip=`hostname -I | cut -d " " -f1`
+      sed -i.bak  's/EF_OUTPUT_ELASTICSEARCH_ENABLE: '\''false'\''/EF_OUTPUT_ELASTICSEARCH_ENABLE: '\''true'\''/' docker-compose.yml
+      sed -i.bak -r "s/EF_OUTPUT_ELASTICSEARCH_ADDRESSES: 'CHANGEME:9200'/EF_OUTPUT_ELASTICSEARCH_ADDRESSES: '$localip:9200'/" docker-compose.yml
+      sed -i.bak -r "s/#EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: 'daily'/EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: 'daily'/" docker-compose.yml
+		```
+
+
+  7. Using PSM, point your DSS firewall syslog (RFC5424) at the IP of your ELK cluster, UDP port 5514  (this number can be changed in the logstash/dss_syslog.conf file in the input section at the top)
 
-  7. Using PSM, point your DSS firewall syslog (RFC5424) at the IP of your ELK cluster, UDP port 5514  (this number can be changed in the logstash/ taormina.conf file in the input section at the top)*
+</br>
 
-  8. If collecting IPFix, use PSM point your DSS IPFix flows (flow export policy) at the IP of your ELK cluster, UDP port 9995  (this port number can be changed in the docker-compose file using the EF_FLOW_SERVER_UDP_PORT parameter)*     
+  8. If collecting IPFix, use PSM point your DSS IPFix flows (flow export policy) at the IP of your ELK cluster, UDP port 9995  (this port number can be changed in the docker-compose file using the EF_FLOW_SERVER_UDP_PORT parameter)*
 
-  10. Run
+</br>
+
+  9. Run
 
      If using docker-compose v1 (standalone)
 
@@ -85,22 +113,58 @@ If these do not match your current install, [check one of the other branches](ht
 
      `docker compose up --detach`
 
-  11. From the install directory, load the elasticsearch schema (mappings) for the Pensando DSS Firewall index-pattern using the following cli:
+  </br>
+  **NOTE:** Give it about 5 minutes to start up
+
+  </br>
+
+  10. From the install directory, load the elasticsearch schema (mappings) for the Pensando DSS Firewall index-pattern using the following cli:
+
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_index_template/pensando-fwlog-session-end?pretty' -d @./elasticsearch/template/pensando-fwlog-session-end.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_index_template/pensando-fwlog-create-allow?pretty' -d @./elasticsearch/template/pensando-fwlog-create-allow.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_index_template/pensando-fwlog-empty-delete?pretty' -d @./elasticsearch/template/pensando-fwlog-empty-delete.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_index_template/pensando-fwlog-create-deny?pretty' -d @./elasticsearch/template/pensando-fwlog-create-deny.json
+
+
+
+
+  11. From the install directory, load the elasticsearch index retention settings for the Pensando DSS Firewall index-pattern using the following cli:
 
-     `curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_index_template/pensando-fwlog?pretty' -d @./elasticsearch/pensando_fwlog_mapping.json`
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_ilm/policy/pensando_empty_delete' -d @./elasticsearch/policy/pensando_empty_delete.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_ilm/policy/pensando_create_allow' -d @./elasticsearch/policy/pensando_create_allow.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_ilm/policy/pensando_session_end' -d @./elasticsearch/policy/pensando_session_end.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_ilm/policy/pensando_create_deny' -d @./elasticsearch/policy/pensando_create_deny.json
+				curl -XPUT -H'Content-Type: application/json' 'http://localhost:9200/_ilm/policy/elastiflow' -d @./elasticsearch/policy/elastiflow.json
+
+
 
-  12. Give it about 5 minutes to start up and point your browser to the ip of your ELK cluster, port 5601
+  12. From the install directory, load the Kibana dashboard for syslog:
 
-  13. In Kibana, import ```./kibana/pensando-dss-elk.ndjson``` into your saved objects
+				curl -X POST "http://localhost:5601/api/saved_objects/_import?overwrite=true" -H "kbn-xsrf: true" -H "securitytenant: global" --form file=@/kibana/pensando-dss-10.15.x-syslog.ndjson
+
+  </br>
 
-  14. If collecting IPFix, in Kibana import ```./kibana/kibana-7.17.x-flow-codex.ndjson``` into your saved objects
+  13. From the install directory, load the Kibana dashboard IPFIX:
+
+				curl -X POST "http://localhost:5601/api/saved_objects/_import?overwrite=true" -H "kbn-xsrf: true" -H "securitytenant: global" --form file=@/kibana/kibana-8.2.x-flow-codex.ndjson
+
+  </br>
+
+
+
+  14. Point your browser to the ip of your ELK cluster, port 5601
+
+  </br>
+
+
 
   15. Use basic docker commands, like ```docker ps``` and ```docker logs <container name>``` to view status of how the containers are doing -
 
+  </br>
+
 *NOTE: It could take about 5 mins for visualizations to become populated in both the DSS and IPFix dashboards.
 
-## Support
-If you need help or have questions, you can [email us](mailto:contact-project+pensando-tbd-elastic-pensando-elk-25427733-issue-@incoming.gitlab.com) and we will get back to you as soon as we can
+</br>
 
 ## Support Policy
-The code and templates in the repo are released under an as-is, best effort, support policy. These scripts should be seen as community supported and AMD Pensando will contribute our expertise as and when possible. We do not provide technical support or help in using or troubleshooting the components of the project through our normal support options. Unless explicitly tagged, all projects or work posted in our GitLab repository (at https://gitlab.com/Pensando).
+The code and templates in the repo are released under an as-is, best effort, support policy. These scripts should be seen as community supported and AMD Pensando will contribute our expertise as and when possible. The absolute best (and quickest) way to get help/support is to [file an issue](https://github.com/amd/pensando-elk/issues).  Any other attempts at contact will probably be lost in the ether and you will rarely, if ever, hear back.

docker-compose.yml

@@ -1,20 +1,19 @@
-version: '3.7'
 services:
   # The environment variable "TAG" is used throughout this file to
   # specify the version of the images to run. The default is set in the
   # '.env' file in this folder. It can be overridden with any normal
   # technique for setting environment variables, for example:
   #
-  #   TAG=6.0.0-beta1 docker-compose up
+  #   TAG=8.11.0 docker-compose up
   #
   # REF: https://docs.docker.com/compose/compose-file/#variable-substitution
   #
   # Also be sure to set the ELASTIC_VERSION variable. For released versions,
   # ${TAG} and ${ELASTIC_VERSION} will be identical, but for pre-release
   # versions, ${TAG} might contain an extra build identifier, like
-  # "6.0.0-beta1-3eab5b40", so a full invocation might look like:
+  # "8.11.0-beta1-3eab5b40", so a full invocation might look like:
   #
-  #   ELASTIC_VERSION=6.0.0-beta1 TAG=6.0.0-beta1-3eab5b40 docker-compose up
+  #   ELASTIC_VERSION=8.11.0-beta1 TAG=8.11.0-beta1-3eab5b40 docker-compose up
   #
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
@@ -28,6 +27,7 @@ services:
       - ES_JAVA_OPTS=-Xms4g -Xmx4g
       - xpack.security.enabled=false
       - action.destructive_requires_name=false
+      - path.repo=/usr/share/elasticsearch/backups
     ulimits:
       nproc: 65535
       memlock:
@@ -69,14 +69,14 @@ services:
 
   # ElastiFlow Unified Collector
   flow-collector:
-    image: elastiflow/flow-collector:6.2.2
+    image: elastiflow/flow-collector:7.0.0
     container_name: pensando-elastiflow
     restart: 'unless-stopped'
     ports:
       - target: 9995
         published: 9995
         protocol: udp
-    network_mode: "host"
+    networks: ['elk-stack']
     depends_on: ['elasticsearch']
     volumes:
       - ./data/elastiflow:/etc/elastiflow

docs/index.md

@@ -0,0 +1,36 @@
+---
+title: Pensando ELK analytics documentation
+---
+
+::: {.toctree maxdepth="2" caption="Contents:"}
+overview download run setup stop
+:::
+
+About
+=====
+
+Pensando-ELK is a tool to search and view data that is generated by one
+or more Pensando Distributed Services Switch (DSS). It is a set of
+containers instantiated by docker-compose and Pensando provided
+visualizations and dashboards to allow for flow monitoring, network
+threat detection and security event viewing.
+
+::: {.note}
+::: {.title}
+Note
+:::
+
+[Check the version in the README
+file](https://gitlab.com/pensando/tbd/siem/elastic/elk-pensando/-/blob/main/README.md?plain=0)
+for the branch you are using to verify that the CXOS and PSM sofware
+releases are compatible with the version of pensando-elk you are using.
+:::
+
+Disclaimer
+==========
+
+This software is provided without warranty or guarantee. Support is on a
+best-effort basis via [Gitlab
+Issues](https://gitlab.com/pensando/tbd/elastic/pensando-elk/issues) and
+is licensed under the [Apache License
+2.0](https://apache.org/licenses/LICENSE-2.0)

elasticsearch/policy/elastiflow.json

@@ -0,0 +1,31 @@
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "min_age": "0ms",
+        "actions": {
+          "set_priority": {
+            "priority": 100
+          }
+        }
+      },
+      "cold": {
+        "min_age": "1d",
+        "actions": {
+        	"readonly": {},
+          "set_priority": {
+            "priority": 50
+          }
+        }
+      },
+      "delete": {
+        "min_age": "7d",
+        "actions": {
+          "delete": {
+            "delete_searchable_snapshot": true
+          }
+        }
+      }
+    }
+  }
+}

elasticsearch/policy/pensando_create_allow.json

@@ -0,0 +1,31 @@
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "min_age": "0ms",
+        "actions": {
+          "set_priority": {
+            "priority": 100
+          }
+        }
+      },
+      "cold": {
+        "min_age": "1h",
+        "actions": {
+          "readonly": {},
+          "set_priority": {
+            "priority": 0
+          }
+        }
+      },
+      "delete": {
+        "min_age": "2h",
+        "actions": {
+          "delete": {
+            "delete_searchable_snapshot": true
+          }
+        }
+      }
+    }
+  }
+}