memory

Memory Fabric

IOA Core's memory fabric provides enterprise-grade data persistence with multi-tier storage, encryption, and automated lifecycle management.

Architecture

The memory fabric implements a 4D tiering strategy:

Hot Storage (Immediate Access)

• Purpose: Active conversations and real-time data
• Backend: Redis/in-memory
• TTL: Configurable (default: 1 hour)
• Encryption: AES-GCM at rest

Warm Storage (Recent History)

• Purpose: Recent sessions and analytics
• Backend: SQLite/PostgreSQL
• Retention: 30 days
• Compression: LZ4

Cold Storage (Archive)

• Purpose: Historical data and compliance records
• Backend: S3-compatible object storage
• Retention: 7 years (configurable)
• Encryption: AES-256 with key rotation

Deep Archive (Regulatory)

• Purpose: Immutable regulatory records
• Backend: Glacier/Deep Archive
• Retention: Indefinite
• Immutability: Cryptographic guarantees

Quick Start

Basic Usage

from ioa_core.memory import MemoryFabric

# Initialize
memory = MemoryFabric()

# Store data
await memory.store(
    key="conversation_123",
    data={
        "messages": [
            {"role": "user", "content": "Hello"},
            {"role": "assistant", "content": "Hi there!"}
        ],
        "metadata": {
            "model": "gpt-4",
            "timestamp": "2025-01-28T10:00:00Z",
            "user_id": "user_456"
        }
    },
    tier="hot",
    tags=["conversation", "active"]
)

# Retrieve data
data = await memory.retrieve("conversation_123")

# Search by tags
results = await memory.search(tags=["conversation"])

Advanced Configuration

from ioa_core.memory import MemoryFabric, MemoryConfig

config = MemoryConfig(
    backends={
        "hot": {
            "type": "redis",
            "host": "localhost",
            "port": 6379,
            "password": "your-password"
        },
        "warm": {
            "type": "postgresql",
            "connection_string": "postgresql://user:pass@localhost/ioa"
        },
        "cold": {
            "type": "s3",
            "bucket": "ioa-archive",
            "region": "us-east-1"
        }
    },
    encryption={
        "algorithm": "AES-GCM",
        "key_rotation_days": 90
    },
    lifecycle={
        "hot_to_warm_hours": 24,
        "warm_to_cold_days": 30,
        "cold_to_archive_years": 7
    }
)

memory = MemoryFabric(config)

Storage Backends

SQLite (Default)

Perfect for development and small deployments.

backend_config = {
    "type": "sqlite",
    "path": "./data/ioa_memory.db",
    "max_connections": 10
}

PostgreSQL

Enterprise-grade relational storage.

backend_config = {
    "type": "postgresql",
    "host": "localhost",
    "port": 5432,
    "database": "ioa_memory",
    "user": "ioa_user",
    "password": "secure_password",
    "ssl": True
}

S3-Compatible

Scalable object storage for archives.

backend_config = {
    "type": "s3",
    "endpoint": "https://s3.amazonaws.com",
    "bucket": "ioa-memory-archive",
    "region": "us-east-1",
    "access_key": "your-key",
    "secret_key": "your-secret"
}

Redis

High-performance caching layer.

backend_config = {
    "type": "redis",
    "host": "localhost",
    "port": 6379,
    "password": "your-password",
    "db": 0,
    "cluster": False
}

Encryption & Security

At-Rest Encryption

All data is encrypted using AES-GCM with:

• Unique encryption keys per data item
• Key derivation from master key
• HMAC for integrity verification

Key Management

• Master Key: Derived from environment variable or HSM
• Data Keys: Generated per item, encrypted with master key
• Rotation: Automatic key rotation every 90 days
• Backup: Encrypted key backups with recovery procedures

Access Control

# Role-based access control
await memory.grant_access(
    resource="conversation_123",
    user_id="analyst_789",
    permissions=["read", "search"]
)

# Audit all access
access_log = await memory.get_access_log("conversation_123")

Lifecycle Management

Automatic Migration

Data automatically moves between tiers based on:

• Age: Time since last access
• Size: Data volume thresholds
• Priority: Business-critical vs. archival data

Custom Policies

# Define custom lifecycle rules
policies = {
    "compliance_data": {
        "retention_years": 7,
        "immutable": True,
        "tier_progression": ["hot", "warm", "cold", "archive"]
    },
    "user_sessions": {
        "ttl_hours": 24,
        "tier_progression": ["hot", "warm"]
    }
}

await memory.set_lifecycle_policies(policies)

Monitoring & Observability

Metrics

# Get storage metrics
metrics = await memory.get_metrics()

print(f"Hot storage: {metrics.hot_usage_gb} GB")
print(f"Warm storage: {metrics.warm_usage_gb} GB")
print(f"Cold storage: {metrics.cold_usage_gb} GB")
print(f"Total operations: {metrics.total_operations}")

Health Checks

# Check backend health
health = await memory.health_check()

for backend, status in health.backends.items():
    print(f"{backend}: {status.state}")
    if status.error:
        print(f"  Error: {status.error}")

Performance Monitoring

• Query latency tracking
• Storage utilization alerts
• Encryption/decryption performance
• Migration success rates

Migration & Backup

Data Migration

# Migrate data between backends
await memory.migrate_data(
    source_tier="warm",
    target_tier="cold",
    filter_criteria={"older_than_days": 30}
)

Backup & Recovery

# Create backup
backup_id = await memory.create_backup(
    tiers=["hot", "warm"],
    destination="s3://backups/ioa-memory"
)

# Restore from backup
await memory.restore_backup(backup_id)

Best Practices

Performance Optimization

1. Use appropriate tiers for access patterns
2. Implement proper indexing for search operations
3. Configure connection pooling for database backends
4. Monitor and adjust TTL settings based on usage

Security

1. Use strong master encryption keys
2. Enable key rotation policies
3. Implement proper access controls
4. Regularly audit access logs

Cost Management

1. Configure appropriate retention policies
2. Use compression for archival data
3. Monitor storage utilization
4. Implement data lifecycle automation

Compliance

1. Enable audit logging for all operations
2. Implement data classification policies
3. Configure immutable storage for compliance data
4. Regular compliance audits and reporting