Many businesses derive huge value from Amazon Web Services (AWS), the leading provider of cloud-based resources for computation, storage and many other flexible services. While these services provide essential capabilities that many business users find to be hugely practical for their needs, many may find that keeping on top of accumulated charges is essential. A user’s AWS account console includes a dashboard and optional reports to monitor charges but any business using an operational intelligence platform such as Splunk will want to integrate as much AWS data as possible into their Splunk monitoring operations, with billing data being a priority. This blog presents the essential steps you’ll need to follow to get your billing data into Splunk via the Splunk Add-on for AWS / Splunk App for AWS and begin monitoring your AWS operations with Splunk.
As a first step, you should install the Splunk Add-on for AWS on your Splunk instance: https://splunkbase.splunk.com/app/1876/.
Upon installation, you can navigate to the “Configuration” tab of the Add-on in the Splunk UI. Here you will be required to specify an AWS Identity and Access Management (IAM) user account and role to access and retrieve relevant data inputs. The user account authentication requires the appropriate key ID and secret key. The role must exist or be newly created within the AWS console, and is either auto-discovered from the account or explicitly specified via the Amazon Resource Number (ARN) in the Splunk Add-on. The role must be configured to authorize access to the inputs we want to read into Splunk: in AWS, select “Services” > “IAM” > “Roles”, create a new role e.g. “Splunk_Role”, and edit the trust relationship to include the full ARN for the user account to be used by the Add-on.
Finally, under “Permissions” for the role, edit or create a new policy that allows the relevant actions to be performed. The Splunk docs for the Add-on provide detailed lists of each input type and its required actions, as well as a catch-all policy for any input you might want to obtain with the Add-on:
http://docs.splunk.com/Documentation/AddOns/released/AWS/ConfigureAWSpermissions
Once configured, the user account and role provide the authentication and authorization to let the Add-on retrieve any of the inputs covered by the policy. Inputs are specified under the “Inputs” tab; there are many different types of input that are covered in some detail by the Splunk docs for the Add-on, however in this blog we will focus only on AWS billing.
Billing data is available from the AWS console in the form of a “Cost and Usage Report”. This is generated by selecting the “Reports” option under “My Billing Dashboard”, then “Create Report”. Various options are available such as whether to generate reports on a daily or hourly basis. A valid S3 bucket must be nominated to receive delivery of the report: verification will require an appropriate permissions policy for the bucket, which can be copied and pasted directly from the sample suggested in the console. Finally, “Report path prefix” should be specified to create the reports within a specific folder in the S3 bucket.
Now we are generating the right format of data in AWS, we can configure our Splunk instance to index it. In the Inputs dashboard of the Splunk Add-on for AWS, create a new input of type “Billing (Cost and Usage Report)”. Provide an input name and select the AWS user account, role and S3 bucket. You will also need to specify a “Report Prefix”, this should match the name of the folder within the S3 bucket i.e. the “Report path prefix” in AWS. The interval for collection defaults to once per hour; confirm or edit this value and save the input to begin to index events of sourcetype “aws:billing:cur”.
Now we have our billing report data available directly in Splunk with all necessary fields needed to keep track of where our AWS costs are coming from, if we care to make our own reports, alerts or dashboards from them. To make our lives easier, we can use the Splunk App for AWS: https://splunkbase.splunk.com/app/1274/.
The app requires no additional configuration and provides numerous dashboards giving practical summaries of the many different AWS inputs, including overall AWS topology, security and usage overviews. For our billing data reports, we can go to the “Billing” drop-down and select “Historical Detailed Bills”: this dashboard gives summaries of total cost, cost over time and costs split by AWS service. The dashboard also provides interactive filtering options to focus on billing data from, for example, different accounts or availability zones.
Another useful dashboard we can use is the “Budget Planner”: here, we just need to select “Detailed Billing” as our data source, input a value in USD for our monthly budget and submit to generate a breakdown of how our monthly charges are comparing to our budget.
Splunk and its Add-on/App for AWS forms the ideal way to monitor your AWS costs and review them alongside other AWS data, or even non-AWS costs represented by other Splunk machine data sources. There are details to consider in getting additional AWS inputs into Splunk that are covered in the Splunk Add-on documentation linked above, though hopefully this blog serves to get you started!