top of page

AWS Storage Cost Optimisation Best Practices

Writer's picture: Ian CorballyIan Corbally

Welcome to our guide on AWS storage services and cost optimisation. In this blog, we will explore various aspects of AWS's storage solutions, providing you with the knowledge you need to make informed decisions and optimise your cloud storage costs effectively.


AWS offers a broad range of storage services designed to cater to different needs, from high-performance databases to long-term data archiving. Understanding these services and their respective use cases is crucial for businesses looking to leverage the cloud for scalability, flexibility, and cost-efficiency.


The Importance of Cost Optimisation

While AWS provides powerful storage options, managing costs can be challenging. Effective cost optimisation strategies can lead to significant savings, allowing businesses to allocate resources more efficiently. This guide aims to equip you with the tools and knowledge to achieve optimal cost management in your AWS storage environment.


Overview of AWS Storage Services

AWS offers a comprehensive suite of storage services tailored to different use cases and requirements. Here's a summary of the main AWS storage services discussed, including Amazon S3, EFS, and FSX, highlighting their key features and use cases.


Amazon S3 (Simple Storage Service)

Amazon S3 is one of the most popular and versatile storage services provided by AWS. It is designed for scalability, data availability, security, and performance. S3 is suitable for a wide range of use cases, including:

  • Backup and Restore: Reliable and durable storage for backing up data.

  • Archiving: Cost-effective storage for long-term data retention.

  • Big Data Analytics: Storage for big data analytics applications.

  • Content Storage and Distribution: Hosting and distributing content such as images, videos, and static files.

Key features of Amazon S3 include:

  • Scalability: Automatically scales to meet your storage needs.

  • Durability: Designed for 99.999999999% (11 9's) durability.

  • Security: Offers multiple security and compliance capabilities.

  • Storage Classes: Various storage classes to optimise cost and performance, such as S3 Standard, S3 Intelligent-Tiering, and S3 Glacier.


Amazon EFS (Elastic File System)

Amazon EFS is a scalable file storage service for use with AWS Cloud services and on-premises resources. It is designed to provide simple, scalable, elastic file storage for use with AWS Cloud services and on-premises resources. EFS is ideal for the following use cases:

  • Content Management and Web Serving: Hosting web content and managing media workflows.

  • Home Directories: Providing scalable and secure home directories for users.

  • Big Data and Analytics: Supporting big data analytics workloads that require shared file storage.

  • Database Backups: Storing database backups and providing high availability.

Key features of Amazon EFS include:

  • Elasticity: Automatically scales your file system storage up or down as you add or remove files.

  • Performance Modes: Two performance modes (General Purpose and Max I/O) to optimise for different workloads.

  • Storage Classes: Standard and Infrequent Access storage classes to optimise costs.

  • Integration: Seamless integration with other AWS services.


Amazon FSX

Amazon FSX provides fully managed third-party file systems with the features and performance of widely-used file systems. There are several types of FSX file systems, each optimised for different use cases:

  • Amazon FSX for Windows File Server: Fully managed Windows file system with support for SMB protocol, ideal for Windows-based applications.

  • Amazon FSX for Lustre: High-performance file system for compute-intensive workloads such as machine learning, high-performance computing (HPC), and media processing.

  • Amazon FSX for NetApp ONTAP: Provides the features of NetApp ONTAP, including data deduplication, compression, and thin provisioning.

  • Amazon FSX for OpenZFS: Offers the features of the OpenZFS file system, ideal for workloads that require high throughput and low latency.

Key features of Amazon FSX include:

  • Managed Service: Fully managed by AWS, reducing the operational burden on users.

  • Performance: High-performance file systems tailored for specific workloads.

  • Compatibility: Supports industry-standard protocols and features.

  • Scalability: Automatically scales to meet your performance and storage needs.


In summary, AWS provides a range of storage services that cater to different needs,

from object storage with Amazon S3, to scalable file storage with Amazon EFS, and high-performance file systems with Amazon FSX. Understanding the key features and appropriate use cases for each service can help organisations optimise their storage costs and performance in the cloud.


Choosing the Right Storage Class

Selecting the appropriate Amazon S3 storage class is crucial for optimising both performance and cost. Below is a step-by-step guide to help you make an informed decision.


Step 1: Understand Your Data Access Patterns

The first step in choosing the right storage class is to understand how frequently and quickly you need to access your data. Different storage classes are designed for different access patterns.

  • S3 Standard: Ideal for frequently accessed data. It offers low latency and high throughput performance.

  • S3 Intelligent-Tiering: Suitable for data with unknown or changing access patterns. It moves data between two access tiers (frequent and infrequent) based on changing access patterns.

  • S3 Standard-IA (Infrequent Access): Best for data that is accessed less frequently but requires rapid access when needed. It's cheaper than S3 Standard but charges for retrieval.

  • S3 One Zone-IA: Similar to Standard-IA but stores data in a single availability zone, making it less durable but more cost-effective.

  • S3 Glacier: Designed for long-term archival of data that is rarely accessed. It offers retrieval times ranging from minutes to hours.

  • S3 Glacier Deep Archive: The cheapest option for data that is rarely accessed and can tolerate retrieval times of 12 hours or more.


Step 2: Evaluate Durability and Availability Requirements

Different storage classes offer varying levels of durability and availability. Consider the criticality of your data and choose a class that meets your needs.

  • High Durability and Availability: S3 Standard and S3 Intelligent-Tiering provide 99.999999999% durability and 99.99% availability.

  • High Durability, Lower Availability: S3 Standard-IA and S3 One Zone-IA offer the same durability but lower availability (99.9% for Standard-IA and 99.5% for One Zone-IA).

  • Archival Storage: S3 Glacier and S3 Glacier Deep Archive provide the same high durability but lower availability, suitable for archival data.


Step 3: Consider Cost Implications

Cost is a significant factor when choosing a storage class. Each class has different pricing for storage, retrieval, and data transfer.

  • S3 Standard: Higher cost but no retrieval fees, ideal for frequently accessed data.

  • S3 Intelligent-Tiering: Slightly higher storage cost but no retrieval fees for frequent access tier.

  • S3 Standard-IA: Lower storage cost but retrieval fees apply.

  • S3 One Zone-IA: Even lower storage cost with retrieval fees, but less durable.

  • S3 Glacier and Glacier Deep Archive: Lowest storage costs with higher retrieval fees and longer retrieval times.


Step 4: Use AWS Pricing Calculator

To get a detailed cost estimate, use the AWS Pricing Calculator. Input your expected storage, retrieval, and data transfer needs to compare the costs of different storage classes.


Step 5: Implement and Monitor

Once you've selected the appropriate storage class, implement it in your AWS environment. Regularly monitor your usage and costs to ensure that your choice continues to meet your needs. AWS offers various tools for monitoring and cost management which will be discussed in the Monitoring and Cost Management Tools section.


Using Lifecycle Policies

Amazon S3 lifecycle policies enable you to automate the transition and deletion of objects based on predefined rules. This feature is crucial for optimising storage costs and ensuring that your data management aligns with your organisational policies.


Setting Up Lifecycle Policies

To set up a lifecycle policy in Amazon S3, follow these steps:

  1. Navigate to the S3 Console: Open the Amazon S3 console from the AWS Management Console.

  2. Select Your Bucket: Choose the bucket for which you want to set up the lifecycle policy.

  3. Access Lifecycle Rules: Go to the 'Management' tab and select 'Lifecycle rules'.

  4. Create a New Rule: Click on 'Create lifecycle rule' and provide a name for the rule.

  5. Define Rule Scope: Select whether the rule applies to the entire bucket or specific prefixes/tags.

  6. Add Transitions: Specify when objects should transition to another storage class (e.g., from Standard to Infrequent Access).

  7. Add Expiration: Set up when objects should be deleted permanently.

  8. Review and Save: Review the settings and save the rule.


Example Lifecycle Policies

Here are some common lifecycle policies you might consider:


Transition to Infrequent Access

If you have data that is not accessed frequently but needs to be retained for a certain period, you can set up a policy to transition objects to the Infrequent Access storage class after 30 days.

{
  "Rules": [
    {
      "ID": "TransitionToIA",
      "Status": "Enabled",
      "Prefix": "",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        }
      ]
    }
  ]
}

Expire Old Data

For data that only needs to be retained for a specific amount of time, you can set a policy to delete objects after 365 days.

{
  "Rules": [
    {
      "ID": "ExpireOldData",
      "Status": "Enabled",
      "Prefix": "",
      "Expiration": {
        "Days": 365
      }
    }
  ]
}

Benefits of Using Lifecycle Policies

  1. Cost Optimisation: Automatically move data to cheaper storage classes or delete it when it's no longer needed, reducing storage costs.

  2. Data Management: Ensure that data is stored in the appropriate class based on its usage pattern, improving data management efficiency.

  3. Compliance: Automate data retention policies to comply with regulatory requirements.

By effectively using lifecycle policies, you can manage your storage costs better and ensure that your data lifecycle aligns with your business needs.


Monitoring and Cost Management Tools

Effectively managing and monitoring your AWS storage costs is crucial for optimising your cloud expenditure. AWS offers several tools designed to help you gain insights into your storage usage and costs. Here are some of the key tools and tips on how to use them effectively:


S3 Storage Lens

S3 Storage Lens provides organisation-wide visibility into object storage usage and activity trends. It includes more than 60 metrics to help you analyse storage usage and identify cost-saving opportunities. There are two tiers of metrics: a free tier and an advanced tier. The advanced tier offers more granular insights and longer data retention periods.


Tips for Using S3 Storage Lens Effectively:

  • Enable Advanced Metrics: If you need deeper insights, consider enabling the advanced metrics, which provide up to 15 months of historical data and more detailed granularity.

  • Set Up Custom Dashboards: Customise your Storage Lens dashboard to monitor key performance indicators (KPIs) that are most relevant to your organisation.

  • Use Storage Lens Groups: Aggregate metrics based on object tags, size, or file extension to gain more customised insights.


CloudWatch

Amazon CloudWatch is a monitoring and management service that provides data and actionable insights to monitor your applications, understand and respond to system-wide performance changes, and optimise resource utilisation. CloudWatch can be integrated with S3 Storage Lens for centralised monitoring.


Tips for Using CloudWatch Effectively:

  • Set Alarms: Configure alarms based on metric thresholds to get notified about unusual activity or potential issues.

  • Dashboard Customisation: Create custom dashboards to visualise your key metrics and performance indicators.

  • Automate Responses: Use CloudWatch Events to automate responses to specific changes in your environment, such as scaling operations or notifying the right team members.


S3 Inventory

S3 Inventory provides a flat-file list of your objects and their metadata, which can be used to audit and report on your S3 usage. It helps you manage storage costs by giving you detailed insights into your data.


Tips for Using S3 Inventory Effectively:

  • Regular Reports: Schedule regular inventory reports to keep track of your storage usage and identify unused or underutilised resources.

  • Integration with Athena: Use Amazon Athena to query your S3 Inventory reports for detailed analysis and insights.

  • Lifecycle Policies: Combine S3 Inventory with lifecycle policies to automate data transitions and deletions, optimising your storage costs.


Cost Explorer

AWS Cost Explorer is a tool that enables you to visualise, understand, and manage your AWS costs and usage over time. It provides detailed insights into your cost and usage trends, helping you identify areas where you can cut costs.


Tips for Using Cost Explorer Effectively:

  • Set Budgets and Alerts: Establish budgets and set up alerts to notify you when you approach your spending limits.

  • Analyse Trends: Use the trend analysis feature to identify cost spikes and understand their causes.

  • Resource Tagging: Implement a tagging strategy to categorise and track your AWS resources more effectively.


Trusted Advisor

AWS Trusted Advisor provides real-time guidance to help you provision your resources following AWS best practices. It includes checks for cost optimisation, performance, security, fault tolerance, and service limits.


Tips for Using Trusted Advisor Effectively:

  • Enable All Checks: Make sure all relevant checks are enabled to get comprehensive insights into your AWS environment.

  • Review Regularly: Regularly review the Trusted Advisor dashboard to identify and address potential issues.

  • Follow Recommendations: Implement the recommendations provided by Trusted Advisor to optimise costs and improve performance.


Choosing the right storage class is crucial for balancing cost and performance. AWS offers multiple storage classes, each designed for different access patterns and data retention requirements. Properly selecting the appropriate storage class can lead to significant cost savings.


Implementing lifecycle policies is another effective strategy for managing storage costs. By automating the transition of data between different storage classes, businesses can ensure that their data is stored cost-effectively throughout its lifecycle.


Monitoring and cost management tools provided by AWS, such as AWS Cost Explorer and AWS Budgets, are indispensable for keeping track of storage expenditures and optimising costs. These tools offer insights and alerts that help businesses stay within budget and make necessary adjustments.


Optimising storage costs is not just about reducing expenses but also about enhancing efficiency and ensuring data is stored in the most appropriate manner. By applying the best practices discussed, businesses can achieve a balance between cost and performance, ultimately leading to better resource management and financial savings.

Subscribe for updates

Keep up to date on the newest cloud cost optimisation tips.

Thanks for submitting!

Gradient Background   _edited.png
morfless collaborates on how to calculate ops, continuous integration tools, amazon s2 pricing, aws cloud server pricing
bottom of page