Part of being a responsible software engineer includes the systems engineering process of configuration management. Although backups and access controls are a necessary part of maintaining a secure IT system, relying on these processes for configuration management is inefficient and dangerous.

Example of git log showing Splunk configuration changes

Thankfully, the text-based configuration of Splunk makes it relatively easy to record configuration changes in a source control system. I prefer git, one of the most widely-used source control applications available. And yes, it’s open source!

Git diff (in tig) showing changes made in a particular commit Assuming you know how git works, here are a few tips and tricks that have worked for me:

• Initialize your git repo in the base Splunk installation directory (/opt/splunk by default)
• Add metadata files to gitignore. Refer to the attached gitignore file to get started.
• Use a separate git repo for apps that you install or create (Separaation of concerns / Single Responsibility Principal)
• Commit frequently, and use a good commit message. If a change isn’t committed, or doesn’t have a useful commit message, you don’t really have a proper configuration management process.
• Use tig to page through the git log and diffs. Install with apt-get install tig or yum install tig.
• Follow this process on all of your Splunk nodes - indexers, forwarders, etc.