Splunk — A developer’s perspective

Developer Lab setup for practice

You can download the trial edition of Splunk or the Splunk cloud to get started. Anything which can give results using regex can be used as an input to Splunk. For example: CSV files, JSON, log files. Data can be uploaded to Splunk, or it can be streamed to Splunk

Uploading Data

In enterprise set up, would have been already loaded from the different system logs using Splunk agents.

  1. Source — What is the location of your log or CSV file
  2. SourceType — For the multiple sources (physical-location-of-log@host) you can have one source type

Field Discovery

Splunk can do a field discovery at 2 stages.

  1. Indexing stage — When the data is being parsed, trimmed, and extracted.
  2. Searching stage — From the fields that are searched.
  1. The Fast mode — field discovery is off. The search result is therefore fastest here.
  2. The Smart mode (default) — Selective use for field discovery as a balance between the other 2 modes.
  3. The Verbose mode — field discovery is done for all fields

Selected Fields & Interested fields

Zoom In/Out

You can choose how much time-series data you want to analyze using this data. It also gives you a bird’s eye view of the volume of data you are analyzing by timeline.

Views

There are 4 views of the data seen in the 4 tabs highlighted below.

1. Events Tab

Each event is separately listed, this is also the default view of the search

2. Patterns

This helps in seeing the pattern % in the data.

3. Statistics

This show the data statistical data in tabular form. From here you can drill down to specific problem areas

4. Visualization

This helps plot the statistics in the form of a graph/chart. There are several visualizations that you can choose from. Using special time-series specific command you can plot stats against a particular timeline.

Results View

You can see the results of the query in any of the 3 forms

  1. List
  2. Raw
  3. Table

Splunk Query Language

There are several operators which are supported by SQL

  1. Pipe
| fields +[field1] -[field2]
| dedup [field1] [field2]
|rename [fieldOldName1] [fieldNewName1]
| head [n] — top n events, the default is 10| tail[n] — bottom n events, the default is 10|sort [field1] — sort on the field in asc order|reverse [field1] — sort on the field in desc order
|top [field1] — Data with maximum occurrences appear in top|rare [field1] — Data with minimum occurrence — useful to identify 
anamolies in data
|highlight [field1][field2] — Highlight the occurrence of a particular data|contingency[ field1][field2] — Shows the matrix of the data in tabular form with x axis as field1, Y axis as field2
|stats avg(field1) by field2 — this will find the average by the group in field2|stats count(field1)|stats min(field1)|stats max(field1)|stats sum(field1)

Sample examples of SQL (Splunk query language)

I will demonstrate the typical SQL constructs used. This I will explain for a workflow like login-service, product catalog listing, ordering, log-out service workflow.

Get all logs in login-service. login-service is a microservice running on different nodes.index=platform-idx sourceType=login_service
Search for “Permission Denied” in the login-serviceindex=platform-idx sourceType=login_service "Permission Denied"
Search for “Permission Denied” in the login-service where the principal is not “John”index=platform-idx sourceType=login_service "Permission Denied" principal!=John
Search for “Permission Denied” in the login-service where the principal is not “John Miller”index=platform-idx sourceType=login_service "Permission Denied" principal!="John Miller"
Search all the log files for “ux002345”index=platform-idx sourceType=* üx002345
Search 2 logs of the login_service and logout_service for ”ux002345"index=platform-idx sourceType=login_service OR sourceType=logout_service üx002345
Get the total count of successful logins to the applicationindex=platform-idx sourceType=login_service login-status=SUCCESS | stats count(login-status)
index=platform-idx sourceType=login_service login-process-time| stats avg(login-process_time) by principal
Create a time-series chart than can used to monitor the processing time of a process by batch_name, time-scale=1 day as the process runs twice within the same calendar day.index=platform-idx sourceType=batch_service | timechart span=1d avg(batch_process_time) by batch_name

Knowledge Objects

  1. Help us in in writing complex queries in easy to interpret simple queries.
  2. The knowledge objects are created inside the Setting menu.
  3. Visibility by default is private

1. Data Interpretation

This has 2 parts to it

2. Data Enrichment

Data extracted by Splunk can be enriched using static or dynamic lookups to expedite easy interpretation of the data

Lookup.csv “0,unemployed”
“1,employed”
lookup employmentStatus.csv employeeStatus OUTPUT employeeStatusName

3. Data Normalization

Helps in the normalization of the data for easy categorization later.

Example: The name of the index in enterprise solution follow naming conventions and therefore are little difficult to remember. They can easily be replaced by user defined nameindex=platform-idx sourceType=order-service book-line-item product-id | transaction product_id maxpause<1dCreate a tag 
PLATFORM for "index=platform-idx"
New queryPLATFORM sourceType=order-service book-line-item product-id | transaction product_id maxpause<1d
| rename <filed1> as <newfieldname>
|eval <newfieldname> =coalesce(<oldfield1>, <oldfield2>)
Define event_type "successful_purchase"index=app_idx sourceType=order_service status=200 action=purchaseAny event that can returned by the above search gets an additional user-defined field "eventtype=successful_purchase" even if you are searching for a totally different search.
Get all the logs events for a user from the “check-out” to “payment-failed” for the username “JohnMiller”index=platform-idx sourceType=payment-service username=JohnMiller | transaction startswith="step=check_out" endswith="payment-failed"
Get the time the average time the user spends inside the website to understand which days users prefer to shopindex=platform-idx sourceType=login-service OR sourceType=logout-server | transaction startswith="login_status=SUCCESS" endswith="logut-status=SUCCESS" | timechart span=1d eval(avg(duration) * 1000) as average-shopping-time
Identify all the products sold twice within a minute in the product-catalogindex=platform-idx sourceType=order-service book-line-item product-id | transaction product_id maxpause<1m

5. Data Models

Data models drive the Pivot tool. It enables users of the Pivot to create compelling reports and dashboards without designing the searches that generate them.

Performance Optimization Tips

  1. Scalability

Hunk

Splunk for Hadoop = HUNK.

Conclusion

The intent of this article is to provide a jump start for developers to start using Splunk. Like everything else, practice makes you perfect and the need for business will drive you to explore more in Splunk and build complex queries.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sarada Sastri

Sarada Sastri

Java Architect | MongoDB | Oracle DB| Application Performance Tuning | Design Thinking | https://www.linkedin.com/in/saradasastri/