Splunk — A developer’s perspective

Developer Lab setup for practice

Uploading Data

  1. Source — What is the location of your log or CSV file
  2. SourceType — For the multiple sources (physical-location-of-log@host) you can have one source type

Field Discovery

  1. Indexing stage — When the data is being parsed, trimmed, and extracted.
  2. Searching stage — From the fields that are searched.
  1. The Fast mode — field discovery is off. The search result is therefore fastest here.
  2. The Smart mode (default) — Selective use for field discovery as a balance between the other 2 modes.
  3. The Verbose mode — field discovery is done for all fields

Selected Fields & Interested fields

Zoom In/Out

Views

1. Events Tab

2. Patterns

3. Statistics

4. Visualization

Results View

  1. List
  2. Raw
  3. Table

Splunk Query Language

  1. Pipe
| fields +[field1] -[field2]
| dedup [field1] [field2]
|rename [fieldOldName1] [fieldNewName1]
| head [n] — top n events, the default is 10| tail[n] — bottom n events, the default is 10|sort [field1] — sort on the field in asc order|reverse [field1] — sort on the field in desc order
|top [field1] — Data with maximum occurrences appear in top|rare [field1] — Data with minimum occurrence — useful to identify 
anamolies in data
|highlight [field1][field2] — Highlight the occurrence of a particular data|contingency[ field1][field2] — Shows the matrix of the data in tabular form with x axis as field1, Y axis as field2
|stats avg(field1) by field2 — this will find the average by the group in field2|stats count(field1)|stats min(field1)|stats max(field1)|stats sum(field1)

Sample examples of SQL (Splunk query language)

Get all logs in login-service. login-service is a microservice running on different nodes.index=platform-idx sourceType=login_service
Search for “Permission Denied” in the login-serviceindex=platform-idx sourceType=login_service "Permission Denied"
Search for “Permission Denied” in the login-service where the principal is not “John”index=platform-idx sourceType=login_service "Permission Denied" principal!=John
Search for “Permission Denied” in the login-service where the principal is not “John Miller”index=platform-idx sourceType=login_service "Permission Denied" principal!="John Miller"
Search all the log files for “ux002345”index=platform-idx sourceType=* üx002345
Search 2 logs of the login_service and logout_service for ”ux002345"index=platform-idx sourceType=login_service OR sourceType=logout_service üx002345
Get the total count of successful logins to the applicationindex=platform-idx sourceType=login_service login-status=SUCCESS | stats count(login-status)
index=platform-idx sourceType=login_service login-process-time| stats avg(login-process_time) by principal
Create a time-series chart than can used to monitor the processing time of a process by batch_name, time-scale=1 day as the process runs twice within the same calendar day.index=platform-idx sourceType=batch_service | timechart span=1d avg(batch_process_time) by batch_name

Knowledge Objects

  1. Help us in in writing complex queries in easy to interpret simple queries.
  2. The knowledge objects are created inside the Setting menu.
  3. Visibility by default is private

1. Data Interpretation

2. Data Enrichment

Lookup.csv “0,unemployed”
“1,employed”
lookup employmentStatus.csv employeeStatus OUTPUT employeeStatusName

3. Data Normalization

Example: The name of the index in enterprise solution follow naming conventions and therefore are little difficult to remember. They can easily be replaced by user defined nameindex=platform-idx sourceType=order-service book-line-item product-id | transaction product_id maxpause<1dCreate a tag 
PLATFORM for "index=platform-idx"
New queryPLATFORM sourceType=order-service book-line-item product-id | transaction product_id maxpause<1d
| rename <filed1> as <newfieldname>
|eval <newfieldname> =coalesce(<oldfield1>, <oldfield2>)
Define event_type "successful_purchase"index=app_idx sourceType=order_service status=200 action=purchaseAny event that can returned by the above search gets an additional user-defined field "eventtype=successful_purchase" even if you are searching for a totally different search.
Get all the logs events for a user from the “check-out” to “payment-failed” for the username “JohnMiller”index=platform-idx sourceType=payment-service username=JohnMiller | transaction startswith="step=check_out" endswith="payment-failed"
Get the time the average time the user spends inside the website to understand which days users prefer to shopindex=platform-idx sourceType=login-service OR sourceType=logout-server | transaction startswith="login_status=SUCCESS" endswith="logut-status=SUCCESS" | timechart span=1d eval(avg(duration) * 1000) as average-shopping-time
Identify all the products sold twice within a minute in the product-catalogindex=platform-idx sourceType=order-service book-line-item product-id | transaction product_id maxpause<1m

5. Data Models

Performance Optimization Tips

  1. Scalability

Hunk

Conclusion

--

--

--

Java Architect | MongoDB | Oracle DB| Application Performance Tuning | Design Thinking | https://www.linkedin.com/in/saradasastri/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Masters and Slaves

Primitive Obsession — code smell that hurt people the most

Drupal in a nut shell

Monitoring and Optimizing performance in PostgreSQL and Google Cloud SQL

Perfect guide to understand Data Scientist in Python

I built a HULU Web App Clone

Ten Advantages of Managed Cloud Services Providers

Ten Advantages of Managed Cloud Services Providers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sarada Sastri

Sarada Sastri

Java Architect | MongoDB | Oracle DB| Application Performance Tuning | Design Thinking | https://www.linkedin.com/in/saradasastri/

More from Medium

DevLog: Move an SQL result from one PostgreDB source to another

Demystifying Digital Transformation: As easy as A, B, C, D!

Demystifying Digital Transformation: As easy as A, B, C, D!

Jira Workflow Best Practices — Part II

Azure EventHub, LogicApp and DataVerse