smart-onion

Machine learning and artificial intelligence layer for Security-Onion

General idea:

The general idea in this project is to build a set of services that will automatically analyse the outputs generated by SecurityOnion by using machine learning, anomaly detection and event correlation to detect Cyber related events.

Architecture

The project's architecture is based on the microservices paradigm:

+ backend - Contains all the backend services and components
|    |
|    +---+ python2_7 - Contains all the backend components that MUST run under Python 2.7 (Only services that use the nupic framework)
|    |        |
|    |        +---+ services - Contains all the background services that MUST run under Python 2.7
|    |                 |
|    |                 +--- alerter - The alerter service is responsible for filtering alerts and aggregating alerts and anomalies into Cyber events
|    |                 |
|    |                 +--- configurator - The configurator service is responsible of allowing the various services to get and set their config as well as configuring the scheduler of the system (Zabbix)
|    |                 |
|    |                 +--- metrics_analyzer - The metrics analyser service should use ML, deep learning, NLP as well as biologically constrained machine learning (nupic) and statistical algorithms to detect anomalies or indications of unexpected behaviour (either human or network or other) and return the probability for an anomaly in a given metric
|    |
|    +---+ python3_5 - Contains all the backend components that run under Python 3.5
|             |
|             +---+ services - Contains all the background services that run under Python 3.5
|                 |
|                 +--- metrics_collector - The metrics collector (a.k.a sampler) service is responsible for querying the raw data (from Security Onion's elasticsearch cluster and from helper DB instances if needed) and create metrics and also allow usage of threshold based alerts
|  
+ frontend - Contains all the frontend services responsible for the UX
|    |
|    +---+ ux - Contains all the different components that make up the WebUI of the system (Written in React)
|    |
|    +---+ ux_svc - Contains a service that connects the WebUI to the Backend services and serves as an API to the system

• Each such service will have its own HTTP listener using the Bottle framework.
• Each such service will not depend on any of the other services except for the configurator.
• Each backend service will be developed in Python 3.5.
• Each frontend service will be developed in NodeJS.
• Each such service will be responsible ONLY for its task as listed above.
• Each service is defined as a GitHub project with issues

For more resources on this project see this folder in Google drive (contains relevant information about Numenta's nupic library and few VMs that will be needed to get you started): https://drive.google.com/open?id=1uol2G02WjjUFv614DC3KC6wPoMR6qUZm

For an interactive architectural diagram with more details see this Prezi presentation here: https://prezi.com/p/sehmdjyc4-kv/

Setup

To install the smart-onion you'll need to following steps (at the moment the system can be installed on a single server only): 0. Install an Ubuntu 16.04 server (Create disks for OS, swap, /tmp, /var/log, metrics, models & model_params & anomaly_likelihood_calcs. Preferably store metrics and models and anomaly_likelihood_calcs on fast dists such as SSDs)

1. Install Zabbix from https://www.zabbix.com/
2. Install Python3.5 and Python2.7
3. Install Git
4. Install Graphite
5. Install StatsD
6. Install Kafka (will be needed in future versions)
7. Install python2.7 requirements file using pip
8. Install python3.5 requirements file using pip
9. Clone git repo
10. Create systemD unit files for the various services and enable them (add them to the project -; )
11. Configure StatsD
12. Import Zabbix template and assign it to the local server
13. Configure whatever needed via the configurator hard-coded config
14. Reboot or start the services
15. Put your knees together and pray... <-;