Client Financial Company Aws Backup and Restore Strategy for New Applications

Name and Sector of Client:­

  • Client Name: Client Financial Company
  • Sector: Financial Services and Banking

Services Include:

  • Personal and commercial banking services
  • Loan and EMI solutions

Primary Work:

  • Financial research and market analysis
  • Risk management and credit assessment
  • Management of Customer Financial Records and Transaction Histories
  • Ensuring Regulatory Compliance and Reporting

Problem Faced by Client:

Client Financial Company completed the migration of their production workloads from on-premises to AWS. After successfully concluding User Acceptance Testing (UAT), they are now focused on deploying five newly developed applications in their AWS production environment. These applications, which handle 32 TB of financial data, require a robust and compliant backup and restore strategy to ensure data protection and regulatory compliance.

Currently, the five newly developed applications are in the testing environment. Even though the data is in the testing environment, proper backup and restore processes are still necessary for production.

The 32 TB of data includes a mix of customer data (personal details, financial profiles, loan records, customer service interactions), transactional data (financial transactions, market trading activities, investment portfolio adjustments, payment processing), regulatory and compliance data (anti-money laundering documentation, Know Your Customer processes, audit trails), and market and economic data (market analysis, risk assessments, economic forecasts). This data is distributed across 5 EC2 instances (2 TB) and 3 RDS databases (10 TB each).

The Applications Overview:

  1. Cross-sell: Enhances sales by recommending additional products based on customer data and preferences. Ensures effective cross-selling strategies are supported by real-time data.
    • Data Criticality: High to Medium
    • Backup Frequency: Every hour
    • Retention Period: 12 hours
    • RTO: 2 hours
    • RPO: 1 hour
  2. Inventory Management System: Tracks inventory levels, orders, and stock movements to maintain optimal stock levels and streamline warehouse operations. Supports real-time updates to prevent stockouts and overstocking.
    • Data Criticality: High to Medium
    • Backup Frequency: Every hour
    • Retention Period: 12 hours
    • RTO: 2 hours
    • RPO: 1 hour
  3. Agent payment: Processes and manages financial transactions related to agent commissions and payments. Ensures accurate and timely disbursements to agents.
    • Data Criticality: High to Medium
    • Backup Frequency: Every hour
    • Retention Period: 12 hours
    • RTO: 2 hours
    • RPO: 1 hour
  1. Customer Assistance Management: Manages customer support requests and tracks resolution progress to enhance customer satisfaction. Supports efficient handling of customer inquiries and issues.
  • Data Criticality: High to Medium
  • Backup Frequency: Every hour
  • Retention Period: 12 hours
  • RTO: 2 hours

    5. Vendor Assistance Management: Oversees interactions with vendors and manages support-related activities. Ensures smooth coordination and issue resolution with vendors.

  • Data Criticality: High to Medium
  • Backup Frequency: Every hour
  • Retention Period: 12 hours
  • RTO: 2 hours
  • RPO: 30 min

Client Financial Company is seeking AWS-native solutions to support the production deployment and protection of these new applications while ensuring full compliance with financial regulations regarding backup and restore procedures.

Challenges for Backups and Restore:

  • Limitation on taking Custom Backups: The current backup infrastructure struggles with scalability, limiting the number of backups that can be efficiently managed for compute and database services.
  • Limited Retention Period: The retention period for snapshot-based backups is restricted, which affects the efficiency and reliability of the backup process.
  • Maintenance Complexity: Manually managing backups across diverse data types is complex and resource-intensive, driving up operational costs.
  • Time-Consuming Restores: Restoring 32 TB of data is time-intensive, leading to potential extended downtime and impacting business operations.

    Proposed Solution & Architecture:

    • AWS Storage Solutions:
    • Amazon S3 (Simple Storage Service): Implement S3 for scalable object storage, providing a robust solution for backing up large volumes of financial data. S3’s scalability ensures that the backup system can grow alongside the client’s data needs.
    • Amazon EBS (Elastic Block Store): Utilize EBS for block-level storage, offering reliable and secure for backup solution. EBS provides high-performance storage that is for handling transactional and regulatory data.
    • Amazon RDS (Relational Database Service): Automate database backups using RDS with the PostgreSQL engine, with using AWS Backup. This ensures reliable and consistent backups of the client’s database systems for financial transactions and records.
    • Automated Backup Management:
    • AWS Backup: As part of the automated backup management solution, AWS Backup provides a centralized and automated backup service that supports various AWS services, including RDS. It offers centralized, policy-based management and compliance features, reducing complexity and ensuring regulatory adherence.
    • Lambda Functions: Deploy Lambda functions to complement AWS Backup by automating the backup process across multiple EBS volumes and virtual machines. This automation enhances flexibility and control, providing custom automation and event-driven backup processes, and ensuring that all data is consistently and securely backed up.

    Reasons for Not Using AWS Backup for EC2 Backups:

    1. Selective File-Level Backups: AWS Backup typically operates at the volume or instance level, but FinTech Innovations requires the ability to back up specific files or directories more frequently, which is not directly supported by AWS Backup.
    2. AMI Creation Limitation: AWS Backup does not directly create an Amazon Machine Image (AMI) from a backup. Instead, it offers the option to “Create an Image from the latest restore point” during a restore operation, which then creates an AMI. This approach might not align with FinTech Innovations’ need for direct and immediate AMI creation.
    3. Granular Scheduling: AWS Backup supports daily, weekly, or monthly backup schedules, but FinTech Innovations requires more frequent backups, such as every 45 minutes, for certain data. AWS Backup’s minimum offering for some resources is 1-hour intervals, which may not meet these specific needs.
    • Data Protection and Security:
    • Key Management Service (KMS): Implement KMS to encrypt all data stored in EBS volumes and S3, ensuring that financial data remains secure. This adds an additional layer of protection, safeguarding against unauthorized access.
    • Cross-Region Replication: Set up cross-region replication for backups, ensuring that data is stored in multiple geographical locations. This redundancy increases data availability and resilience against regional failures or disasters.
    • Efficient Data Restoration:
    • Secrets Manager Integration: Used AWS Secrets Manager to securely manage access credentials during the restoration process, ensuring that only authorized personnel can restore data.
    • Primary Region Restore: Restores data from S3 to Lambda functions for handling regular image updates in an auto-scaling group. This process ensures that the latest images are quickly and efficiently distributed across instances, maintaining optimal performance and availability in the primary region.
    • Secondary Region Restore: Manages data restoration from S3 through Lambda functions to various components including Load Balancer (LB), Auto Scaling Group (ASG), EC2 instances, Route 53, Web Application Firewall (WAF), and NAT gateway. This comprehensive approach ensures that all necessary services and infrastructure are promptly and accurately reconfigured, ensuring continuity and resilience in the secondary region.
    • Regulatory Compliance:
    • Ensured that all backup and restoration processes adhere to financial regulations and internal policies. Implement necessary checks and balances to maintain data security and accuracy throughout the backup and restoration lifecycle.
    • IAM and Bucket Policies for enhanced security and control in accessibility to avoid any vulnerability that may arise

    • General Architecture:

      Architecture-1: (EC2 Application Backup and Restoration Architecture)

      Architecture-2: (RDS Data Backup and Restoration Architecture)

      Architecture-3: (S3-Life-Cycle-Policy)

Outcomes:

  1. Increased Data Scalability and Storage Efficiency:

1.1. Amazon S3:

The implementation will manage 32 TB of financial data efficiently, with the capability to scale as data volumes increase. This eliminates the previous storage constraints and enhances the ability to store additional data seamlessly.

  1. Enhanced Data Security:

2.1. AWS Key Management Service (KMS) & Cross-Region Replication:

With encryption applied to all stored data and secure replication across multiple regions, the solution will significantly reduce the risk of data breaches and ensure that data is protected against regional failures.

  1. Operational Efficiency through Automation:

3.1. AWS Lambda:

Automating backup and restore processes will minimize manual intervention, reducing the time spent on these tasks and allowing for more efficient resource use. The system will automatically handle most backup tasks.

  1. Reliable and Fast Data Restoration:

4.1. Automated Restore Processes & AWS Secrets Manager:

Automated restore processes and secure credential management via AWS Secrets Manager will ensure quick and accurate data restoration, minimizing downtime and enhancing business continuity during disaster recovery scenarios.

  1. ­Increased Data Resilience and Availability:

5.1. Cross-Region Replication:

Replicating data across several AWS regions will increase its availability and resilience, providing continuous access to data even in the event of a regional outage.

  1. Compliance with Internal Policies and Best Practices:

6.1. AWS Compliance Tools:

The solution will ensure adherence to internal data governance policies and best practices, reducing the risk of non-compliance and aligning with audit requirements.