encryption_and_vulnerability_check

This Splunk app checks for Splunk Vulnerability Disclosures and gives you an overview of your environment's encryption status.

How to install

• Download all apps. The TAs are supported under Linux only.
• The app should work in Windows environments but you won't get all data and results from all dashboards/use cases.
• Depending on how you've set up your indexes.conf [default] stanza, you may need to adjust the indexes.conf found under TA-indexer_eavc\default\. If you are using volume definitions for indexes, you may want to adjust the default values found inside the add-on.

Place the apps on your servers the following way:

Name	Deployed on	Description
app_eavc	One Search Head of your choice	This app contains most of the searches, all dashboards and all of the lookups. It's not recommended to put this app onto a Search Head with a Premium App (ES/ITSI).
SA-mc_eavc	On your Monitoring Console	This Supporting Add-on contains a couple of REST searches that only can be ran on the MC to provide complete results. These are written to the app's index (app_eavc). All data will only be available when the MC is configured and roles are assigned to all Splunk instances.
TA_eavc	Goes onto your Splunk Enterprise instances, except indexers	contains a few scripts that will do checks for all instances
TA-indexer_eavc	Cluster Manager/manager-apps or on an Indexer if not clustered	Contains scripts with links to peer-apps instead of apps. Just small linux commands to query some OS-related checks, no heavy stuff. Also contains the app's index definition (app_eavc).

Example Architecture

Name	Function	Will get
SH1 (users)	Search Head (no premium apps), for all users	TA_eavc ($SPLUNK_HOME/etc/apps)
SH2 (admin)	example administrative SH, also hosts the Deployment Server. This one gets the EAVC dashboards in this example	app_eavc, TA_eavc ($SPLUNK_HOME/etc/apps)
SH (CM/LM/MC)	Cluster Manager, License Manager, also hosting the Monitoring Console	TA_eavc, SA-mc_eavc ($SPLUNK_HOME/etc/apps/) and TA-indexer_eavc ($SPLUNK_HOME/etc/manager-apps/)
IDX1	cluster peer	TA-indexer_eavc ($SPLUNK_HOME/etc/peer-apps/)
IDX2	cluster peer	TA-indexer_eavc ($SPLUNK_HOME/etc/peer-apps/)
IDX3	cluster peer	TA-indexer_eavc ($SPLUNK_HOME/etc/peer-apps/)
SH-ES	Enterprise Security Search Head	TA_eavc ($SPLUNK_HOME/etc/apps)
SH-ITSI	IT Service Intelligence Search Head	TA_eavc ($SPLUNK_HOME/etc/apps)

If you have a SH that is solely used for the Monitoring Console or the Deployment Server (so, no Cluster Manager), you may consider using the app on that Search Head to avoid having it take resources away from other SHs that may have heavy load. It's up to you.

Use Case Overview by Operating System

Name	OS	Description	Use Cases
app_eavc	Linux, Windows	Will give you an overview of the SVDs. Most SVDs being able to be validated simply by checking your Splunk Enterprise and UF versions (as well as other Splunk official Apps installed).	Splunk Security Advisories Mitigation (incomplete), Splunk Supported Versions (incomplete), Hardening Report (incomplete)
SA-mc_eavc	Linux, Windows	Will populate the app_eavc index with data from REST searches. Grants additional insight into some SVDs and provides the data for many additional dashboards and use cases (e.g. Hardening Report).	Splunk Security Advisories (incomplete), Splunk Supported Versions (complete), Encryption Status (incomplete), Hardening Report (incomplete)
TA_eavc	Linux	Scripts for Operating System checks. Some hardening and best practice checks to further enrich the results of the Hardening Report and other dashboards. Scripts found here will only work for Linux-based instances.	Enriches missing data for all use cases/dashboards.
TA-indexer_eavc	Linux, Windows(*)	Deploy the index configuration (Windows + Linux). Scripts found here will only work for Linux-based instances.	Needed to store data for complete use cases and dashboard data. Also completes use cases with indexer hardening/OS checks.

Getting started after installation

After all apps were installed, you will not see results immediately. Some reports may be delayed, a few are disabled because they usually only need to run once. All steps except the last step 8 are optional, as they will run after some time. I do not recommend running the reports of the app before the TAs were deployed to all Splunk Enterprise instances because data will be missing and not being re-generated. All SA-MC reports will check whether data is already in that index+sourcetype and not populate it again if data was found.

1. Run the reports on your Monitoring Console first if they're not run by default. There are several reports starting with "SA-mc_eavc_*"
2. Go to your Search Head, select the Encryption and Vulnerability Check App
3. Select Additional Links > Reports
4. Here, you will find a Universal Forwarder report for querying versions below 9.2 and one for versions starting 9.2. Select one appropriate to your environment. More on that later. Only one should be scheduled (pre_9.2 is scheduled by default).
5. run query_splunk_version
6. run populate_svd_upgrade_checks, populate_svd_uf_upgrade_checks, populate_svd_app_upgrade_checks
7. TA_eavc contains scripts to get certain OS checks done. These may take a while to populate as well but usually don't affect the SVD status.
8. Run populate_manual_checks_lookup once. This one is disabled and only needs to be run once. Afterwards, that lookup has to be manually edited because not all checks can be done automatically (mostly SOAR-related SVDs).

• The app (and only the app_eavc) contains scripts which connect to this repository. They check for 1. new Splunk Vulnerability Disclosures (SVDs) and 2. other lookups of app_eavc (like Splunk supported product versions). That way you don't have to upgrade the whole app everytime new vulnerabilities have been made public. If there's no connection, you just won't get updated lookups automatically.
• Starting with Splunk version 9.2, Splunk will log Universal Forwarder related data in the internal index _dsclient which is used for version comparison. This is also the only method if something like Cribl (or Heavy Forwarders) sits between Splunk Enterprise and the Universal Forwarders.
• The time ranges on dashboards may need to be adjusted to your needs, because over time, the data will fade out of the default ranges. Default is often searching the last 7 days, while the data is kept for 14 days. Only once the data is purged by Splunk, the generating reports will repopulate the sourcetypes again.
• I'd recommend to allow the SH to connect to raw.githubusercontent.com for the automatic updates of lookups to stay ahead of future Splunk vulnerabilities
• The SA-mc reports populating the app_eavc index run usually only once every few days because they will check whether data is found in that index. If you're having changes to your apps, you might want to consider deleting the data and letting the report run again. The indexed data will be kept for 14 days.
• For easy clearing of the SA-mc data, there is a report called SA-mc_eavc_purge_sourcetypes, which will only delete sourcetypes starting with eavc:mc_*. The | delete is inside a comment, so you can safely open the search.
• Inside the Encryption and Vulnerability Check app, you either need to be admin or grant you the r_encryption_check role in order to see all extracted fields and other Knowledge Objects.

This app contains the https://github.com/drwetter/testssl.sh script to check the servers for vulnerabilities and other checks against the running webserver, as Splunk doesn't provide much of an insight into the webserver details. The data might be coming in already but I'm still figuring out how to use that data in dashboards/use cases. Consider the data optional for now.

For non-clustered Indexers

If you want to test the app on a non-clustered indexer, you can still use the TA-indexer_eavc. The inputs.conf has all inputs in there. Standard enabled inputs are for $SPLUNK_HOME/etc/peer-apps/ but it also contains the same scripts with links to $SPLUNK_HOME/etc/apps in comments.

Version 3

Version 3 was a complete refactoring of the existing app. The most important changes are that the app that used to exist now is divided into an app, a Supporting Add-on and two Technical Add-ons. Next to the refactoring for bigger environments, many new reports, dashboards, lookups and use cases were implemented.

Version 2

Version 2 has its dashboards migrated to the Dashboard Framework with some changes and additions. Due to limits of the Dashboard Framework's features, drilldowns with tokens are limited, unfortunately.

Biggest changes are the implemented checks for the Splunk Vulnerability Disclosures. These lookups can be updated with the script found under bin (enabled by default). The script runs every 30 minutes and checks for changes on github.

The app contains a few scheduled searches. If you update the lookups OR use the app for the first time, the mitigation status of the SVDs might take a while to update. If you want to fasten things up, run the reports found in the app.

For RBAC, you can use the shipped read-only or writeable role of the app. The app does not contain or need any additional index.

New in version 2.3.1 is a report, which can be ran manually to populate a manual checks lookup. After running this, check the lookup in order to solve the manual checks displayed in the Splunk Security Advisories dashboard.

The future / upcoming versions

Version 3.1.0 is planned to have more use cases (especially for the Hardening Report dashboard) with more checks. Other searches may be further optimized and more reports added to the MC. Don't worry, there will never be heavy load on the MC. Just simple REST searches to query all Splunk Enterprise instances from a cluster every few hours. Also, the Hardening Report ues cases will be put into groups (eg. SH, indexer, best practices, encryption etc.). Furthermore, the Hardening Report does not contain the recommendations on how to fix the failed checks right now.

As of now, the Search Head app (app_eavc) does not contain any alerts. I plan to include (disabled) alerts, e.g. for new vulnerabilities which aren't fixed in the current version. That way you don't have to check the dashboards preemptively.

Version 3.2.0 will undergo another refactoring to rename many of the saved searches, which will help with new installations/setups. As the number of reports keeps growing, it's hard to keep track of what to run first. The scheduled times are set very rarely due to the function of the app: most searches just need to run once and then it's fine if they don't run again for hours or even days.

In the far future, I plan to deploy my environment on Kubernetes, so I will be doing Kubernetes-related checks at one point, but this definitely won't happen in 2024.

About the Content/Use Cases

You may be wondering "what's in it?". I'd say, take a look yourself. It all started as a debugging app with version 1 with some dashboards displaying the status of encryption parameters and some hardening stuff. Version 2 was a huge advancement to include the matching of Splunk's Vulnerability Disclosures mapped to your environment (what needs to be fixed and what not). Version 3 was heavily tested for distributed bigger environments and thus needed some changes, making a single app impossible to use for clusters. One could argue to put the app on the MC but the use cases are becoming more and more, so I divided the app into seperate logical parts; getting data by REST, getting data by scripts, indexer related configs and the visuals+reports (app). The checks for multi-site environments should work but still need to be verified.

Here's a rough estimate what to find in the app:

• a full list of Splunk Vulnerability Disclosures (SVDs) with a mitigation status for each SVD

  • the SVDs Mitigation Status includes Core, ES, ITSI and other apps that may be installed in your environment

• Encryption Overview: display several important parameters for TLS-secured environments and other Splunk-specific configs you might not even know of

• the Splunk Server Status dashboards contain a few dashboards displaying a lot of stuff: forwarder status, indexer, webserver settings, specific Linux server use cases (who owns $SPLUNK_HOME, directory permissions, Kernel version)

• Splunk Supported Versions: lists the End Of Life for Splunk Enterprise, SOAR, UBA, Enterprise Security and IT Service Intelligence and maps it to your environment when ES/ITSI are found

• Hardening Report: this is basically a list of growing use cases that are focussed on security-related stuff and hardening of your whole Splunk environment. Don't expect to get all checks passed. I've been doing this for many years and I'm very strict. As of app verson 3.0.7, 25 checks are included.

Upcoming are other checks (you may get a hint from other dashboards), but the most important focus will be adding more hardening and best practice checks to the Hardening Report, keeping the SVDs up to date, as well as automating the updating of included lookups. I'd like to install this app once, set it up once and then don't care about manual app updates again.

Issues

• Unfortunately, github doesn't want to recognise the executable flags of my scripts. So you may need to add the executable flags to the scripts to get all of the data. Finally fixed the permissions again. Let's hope it stays that way.
• Splunk does not provide a downloadable list (or an API) to get all SVDs without effort, so I have to grep them from their website and format them in a way that I can use them in this app. This may change in the far future.

Restrictions

• Currently, the SVD checks rely on having the same Splunk Enterprise version across your cluster. If I would take into account having multiple Core versions in the same environment, the checks would be way more complicated. The checks should work with different versions though but it's not verified for Splunk Enterprise. Universal Forwarder checks do take multiple versions into account though. Those use cases were built like this because it's common to have multiple UF versions in the infrastructure, sometimes including really old ones.
• This app was developed and tested for Linux environments only. I didn't see a Windows-based Splunk Enterprise installation in years and I'm no Windows expert. So, there won't be any Windows Server related checks anytime soon. Also, even if possible, this would require way more add-ons and a further separation of scripts.