storage-provider

Instruction file imported from cycle123582/label-studio-Chinese (.cursor/rules/storage-provider.mdc). Copyright stays with the author.

Implementing New Storage Providers in Label Studio

Overview

This document describes the process and best practices for adding a new storage provider to Label Studio using the declarative provider schema system.

Label Studio supports 2 types of cloud storages:

1. Import Storages (Source Cloud Storages) - for importing tasks/data
2. Export Storages (Target Cloud Storages) - for exporting annotations

See comprehensive overview about storages @io_storages/README.md.

Implementation Checklist

Follow all steps below to implement a new storage. More details follow after the checklist; review them all. Do it on your best, until all items are done and tests are passing.

1. Exploration and preparation

1. Carefully read @io_storages/README.md
2. Search official documentation for the new storage you want to add

• Determine whether pre-signed URLs are supported, or whether only direct reads are possible. In case of direct reads, we should hide pre-signed URLs toggle and use Label Studio proxy.
• Determine whether writes are supported, and how annotations will be stored (objects/blobs, files, rows/strings in a table, etc.)
• Understand the provider's Python API/SDK, especially how to read, write, and list objects. If SDK is available, use SDK

3. If the requester hasn't specified the target edition, recommend Open Source or Enterprise and confirm the choice
4. Check storage directory structure in label_studio/io_storages (or label_studio_enterprise/lse_io_storages for Enterprise) and the s3 (or s3s for Enterprise) subfolder
5. Create the new provider directory structure based on the pattern you observed
6. Create a README.md file in the new provider folder
7. Add a brief Overview section about the new storage and your findings from step 2
8. Add a section on how to configure the storage from scratch for users unfamiliar with it. Provide clear, correct, up-to-date steps with links to official docs to reduce manual QA time

2. Backend Implementation

1. Implement storage mixin with common fields:

• Basic fields: bucket, prefix, regex_filter, use_blob_urls (pre-signed URLs on/off), recursive_scan (if applicable)
• URL resolution: presign, presign_ttl (if applicable to the storage)
• Provider credentials: api_key, secret_key, endpoint_url
• Common methods: get_client(), validate_connection()

2. Create import storage base class with required methods:

• iter_objects() - iterate over storage objects
• get_data() - load task data from objects
• generate_http_url() - create HTTP URLs
• can_resolve_url() - check URL resolution capability
• validate_connection() - validate credentials and that the prefix contains files

3. Create export storage class with required methods:

• save_annotation() - save single annotation to storage
• delete_annotation() - delete annotation from storage (optional)
• validate_connection() - validate credentials and bucket access (NO prefix check)

4. Create non-abstract provider-specific concrete classes for import and export
5. Implement storage link models:

• ImportStorageLink for tracking task imports
• ExportStorageLink for tracking annotation exports

6. CRITICAL: Add app_label = 'io_storages' to Meta classes - All concrete storage models (ImportStorage, ExportStorage, and StorageLink classes) must include app_label = 'io_storages' in their Meta class to avoid Django app registration errors. This is required because storage providers are in subdirectories of io_storages but need to be registered under the main io_storages app. Note: Enterprise providers do NOT need app_label - see enterprise guide.
7. Create serializers with validation logic
8. Implement API views following existing patterns
9. Register URLs in storage URL configuration
10. Add signal handlers for auto-export functionality:

• post_save signal for automatic annotation export
• pre_delete signal for automatic annotation deletion
• Async export functions with error handling

11. If you use SDK: add provider SDK library to pyproject.toml

• Make poetry lock: poetry install && poetry lock

12. Create database migrations using poetry run python manage.py makemigrations only!
13. Ensure that you correctly handle token and security fields; they should not be displayed on the frontend or backend after they are initially entered and saved. Verify how this works with other storage codes.

3. Frontend Implementation

1. Check examples: for Open Source see: label-studio/web/apps/labelstudio/src/pages/Settings/StorageSettings/providers/, for Enterprise see: label-studio-enterprise/web/apps/labelstudio/src/pages/Settings/StorageSettings/providers/
2. Create a provider configuration file in web/apps/labelstudio/src/pages/Settings/StorageSettings/providers/ with:

• All required fields with proper types
• Zod validation schemas
• Meaningful labels and placeholders
• Proper field layout definition

3. Register provider in central registry
4. Mark credential fields with accessKey: true
5. Test form rendering and validation
6. Verify edit mode behavior for credentials

4. Testing

• Write backend pytests for newly added API calls (see @backend-unit-tests.mdc for details)
• Test connection validation (validate_connection)
• Test object iteration and filtering (iter_objects)
• Test task data loading (get_data)
• Test frontend form functionality
• Test export annotations on Sync button click and when Submit button clicked (post save signal)
• Test delete exported annotation
• Critical: run all created tests, check how to run them in @backend-unit-tests.mdc

5. Documentation

• Add provider to storage documentation (docs/source/guide/storage.md)
• Update API documentation using @method_decorator for storage API classes (see @updating-label-studio-sdk.mdc)

6. Git

• Commit all added and modified files related to the new storage into git, use git add <specific files> and never use git commit -a.

7. Integration & Deployment

These steps are for manual QA by the requester; remind them after you finish your work:

• Test end-to-end storage workflow
  • Create a project, add a new import storage, sync it, and check Data Manager for new files
  • Create a project, add a new export storage, create a few annotations, sync the storage, and check that files appear in the storage admin console
• Test URL resolution: verify that storage URIs like s3://xxx/1.jpg are resolved and load in the editor
• Test with both presigned URLs and proxy mode
• Test storage error and status reporting: if there are any errors, a user should be able to click "Failed - View logs" and see an error description

Decision: Open Source vs Enterprise

CRITICAL FIRST DECISION: Where should your new storage provider be implemented?

Add to Open Source (io_storages) if:

• Basic authentication (API keys, service accounts)
• Standard file formats (JSON, JSONL, images)
• Community-focused features
• Simple cloud storage connectivity
• User request: the requester explicitly asks for Open Source

Add to Enterprise (lse_io_storages) if you need:

• Advanced Authentication: IAM roles, Workload Identity Federation, cross-account access
• Enterprise Security: Server-side encryption, ACL controls, audit logging
• Advanced Data Formats: Parquet support, complex data transformations
• Billing Restrictions: Storage limits, organizational constraints
• Advanced Review Workflows: Review-based export triggers
• User request: The requester explicitly asks for Enterprise

Key Enterprise Advantages

1. No App Label Issues: LSE uses proper app configuration, avoiding Django registration conflicts
2. Advanced Authentication: Support for IAM roles, WIF, cross-account access
3. Enhanced Security: Built-in encryption, ACL controls, audit capabilities
4. Enterprise Features: Parquet support, billing controls, review workflows
5. Better Error Handling: Enhanced logging, metrics, and monitoring
6. Scalability: Client caching, optimized performance patterns

Default Recommendation: Most new storage providers should be added to Enterprise for better security, features, and future extensibility.

Backend Implementation

Important note: if you implement a storage for Label Studio Enterprise, replace all paths label_studio/io_storages with label_studio_enterprise/lse_io_storages.

1. Create Storage Models

File Structure

Create these files in label_studio/io_storages/yourprovider/:

• __init__.py
• models.py - Core storage models
• serializers.py - API serializers
• api.py - API views
• utils.py - Provider-specific utilities
• form_layout.yml - Form layout (optional, for compatibility)

Storage Mixin Pattern

Create your storage mixin in label_studio/io_storages/yourprovider/models.py:

Reference Implementation: Follow io_storages/s3/models.py or io_storages/gcs/models.py

Required Fields:

• Common: bucket, prefix, regex_filter, use_blob_urls
• URL Generation: presign, presign_ttl
• Credentials: Provider-specific fields (api_key, secret_key, endpoint_url)

Key Methods to Implement:

• get_client(): Initialize provider client (see S3StorageMixin.get_client())
• validate_connection(): Validate credentials and bucket access
  • Import storages: Must check prefix contains files
  • Export storages: Only validate bucket (prefix auto-created)

Important: Make your mixin abstract with class Meta: abstract = True

Import Storage Classes

Reference Implementation: Follow io_storages/s3/models.py classes:

• S3ImportStorageBase for the base functionality
• S3ImportStorage for the concrete implementation

Required Classes:

class YourProviderImportStorageBase(YourProviderStorageMixin, ImportStorage):
    """Abstract base - implement core import methods"""
    
class YourProviderImportStorage(ProjectStorageMixin, YourProviderImportStorageBase):
    """Concrete class for database storage"""

Key Methods to Implement:

• iter_objects(): Iterate storage objects (see S3ImportStorageBase.iter_objects())
• get_data(key): Parse task data from storage objects (supports blob URLs and JSON parsing)
• generate_http_url(url): Generate accessible URLs for data (presigned or proxy)
• can_resolve_url(url): Check if URL matches your provider's pattern

Export Storage Class

Reference Implementation: Follow io_storages/s3/models.py S3ExportStorage class as an example

Required Class:

class YourProviderExportStorage(YourProviderStorageMixin, ExportStorage):
    """Concrete export storage - inherits bulk methods from ExportStorage"""

Key Methods to Implement:

1. save_annotation(annotation) - Save single annotation (see S3ExportStorage.save_annotation()):

• Use self._get_serialized_data(annotation) for format-aware serialization
• Use YourProviderExportStorageLink.get_key(annotation) for key generation
• Handle provider-specific upload logic
• Create storage link to track export: YourProviderExportStorageLink.create(annotation, self)

2. delete_annotation(annotation) (Optional) - Delete from storage (see S3ExportStorage.delete_annotation()):

• Check self.can_delete_objects flag first
• Use same key generation as save method
• Remove storage link after successful deletion

Inherited Methods: save_annotations() and save_all_annotations() are inherited from ExportStorage with built-in parallel processing and progress tracking.

Storage Link Models

Reference Implementation: Follow io_storages/s3/models.py link classes

Required Classes:

class YourProviderImportStorageLink(ImportStorageLink):
    storage = models.ForeignKey(YourProviderImportStorage, on_delete=models.CASCADE, related_name='links')

class YourProviderExportStorageLink(ExportStorageLink):
    storage = models.ForeignKey(YourProviderExportStorage, on_delete=models.CASCADE, related_name='links')

These classes track which tasks/annotations have been imported/exported and provide key generation for storage objects.

Signal Handlers for Automatic Export

Reference Implementation: Follow io_storages/s3/models.py signal patterns

Required Signal Handlers (add to your models.py):

1. @receiver(post_save, sender=Annotation) - Auto-export when annotations created/updated

   • Check if project has export storages configured
   • Use start_job_async_or_sync() to handle export asynchronously

2. @receiver(pre_delete, sender=Annotation) - Auto-delete when annotations deleted

   • Check storage.can_delete_objects before deletion
   • Call storage.delete_annotation() for each configured storage

Key Pattern: Use Django's getattr(project, 'io_storages_yourproviderexportstorages', None) to access related storages

2. Create Serializers

Create serializers in label_studio/io_storages/yourprovider/serializers.py:

Reference Implementation: Follow io_storages/s3/serializers.py patterns

Required Classes:

• YourProviderImportStorageSerializer(ImportStorageSerializer)
• YourProviderExportStorageSerializer(ExportStorageSerializer)
• Optional: YourProviderStorageSerializerMixin for shared functionality

Key Features:

1. Security: Define secure_fields = ['api_key', 'secret_key'] to hide credentials in API responses
2. Validation: Override validate() method to call storage.validate_connection()
3. Type Field: Add type = serializers.ReadOnlyField(default=os.path.basename(os.path.dirname(__file__)))

3. Create API Views

Create API views in label_studio/io_storages/yourprovider/api.py:

Reference Implementation: Follow io_storages/s3/api.py patterns

Required API Classes (10 total):

Import Storage APIs:

• YourProviderImportStorageListAPI(ImportStorageListAPI)
• YourProviderImportStorageDetailAPI(ImportStorageDetailAPI)
• YourProviderImportStorageSyncAPI(ImportStorageSyncAPI)
• YourProviderImportStorageValidateAPI(ImportStorageValidateAPI)
• YourProviderImportStorageFormLayoutAPI(ImportStorageFormLayoutAPI)

Export Storage APIs:

• YourProviderExportStorageListAPI(ExportStorageListAPI)
• YourProviderExportStorageDetailAPI(ExportStorageDetailAPI)
• YourProviderExportStorageSyncAPI(ExportStorageSyncAPI)
• YourProviderExportStorageValidateAPI(ExportStorageValidateAPI)
• YourProviderExportStorageFormLayoutAPI(ExportStorageFormLayoutAPI)

Key Features:

1. OpenAPI Documentation: Use @method_decorator with extend_schema for each endpoint
2. Queryset & Serializer: Set queryset and serializer_class for each view

4. Register URLs

Reference Implementation: Follow io_storages/s3/urls.py patterns

1. Update Main URLs - Add to label_studio/io_storages/urls.py:

path('api/storages/yourprovider/', include(('io_storages.yourprovider.urls', 'io_storages'), namespace='yourprovider-api')),

2. Create Provider URLs - Create label_studio/io_storages/yourprovider/urls.py:

Required URL Patterns (10 endpoints):

• Import: /import/, /import/<pk>/, /import/<pk>/sync/, /import/validate/, /import/form-layout/
• Export: /export/, /export/<pk>/, /export/<pk>/sync/, /export/validate/, /export/form-layout/

Pattern: Use api.YourProviderImportStorageListAPI.as_view() for each endpoint

Frontend Implementation

Create Provider Configuration

Reference Implementation: Follow web/apps/labelstudio/src/pages/Settings/StorageSettings/providers/s3.ts

Create: web/apps/labelstudio/src/pages/Settings/StorageSettings/providers/yourprovider.ts

Required Structure:

const yourProviderProvider: ProviderConfig = {
  name: "yourprovider",
  title: "Your Provider",
  description: "Connect to your provider storage",
  icon: IconCloudProviderYourProvider,
  fields: [/* field definitions */],
  layout: [/* field layout */]
};

Key Field Properties:

• accessKey: true: Mark credential fields for secure handling
• schema: Use Zod validation with .default() for defaults
• type: text, password, select, toggle, counter
• Placeholders: Provide realistic examples for all fields

Important:

• Exclude global fields: Don't include title, regex_filter, use_blob_urls
• Credential fields: Use type: "password" with accessKey: true
• Register provider: Add to providers/index.ts export

Reference Examples: See s3.ts, gcs.ts, azure.ts for field patterns and validation schemas

Testing

Reference Implementation: Follow tests in label_studio/io_storages/tests/ and label_studio_enterprise/lse_io_storages/tests/. Useful examples include test_import_storage_list_files_api.py, test_proxy_api.py, and test_get_bytes_stream.py.

Create: label_studio/io_storages/tests/test_yourprovider.py

Required Test Classes:

• TestYourProviderStorage - Basic storage functionality tests
• TestYourProviderImportStorage - Import-specific tests
• TestYourProviderExportStorage - Export-specific tests

Key Test Methods:

• test_connection_validation() - Test credential and connection validation
• test_object_iteration() - Test storage object listing and filtering
• test_data_loading() - Test task data loading from storage objects
• test_export_functionality() - Test annotation export and deletion

Common Issues & Solutions

Django App Label Error

Error: RuntimeError: Model class doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS

Cause: Storage provider models in subdirectories (e.g., io_storages.databricks) are not automatically recognized as belonging to the io_storages app.

Solution: Add explicit app_label = 'io_storages' to all concrete model Meta classes:

class YourProviderImportStorage(ProjectStorageMixin, YourProviderImportStorageBase):
    class Meta:
        abstract = False
        app_label = 'io_storages'  # Required!

class YourProviderExportStorage(YourProviderMixin, ExportStorage):
    # ... fields ...
    
    class Meta:
        app_label = 'io_storages'  # Required!

class YourProviderImportStorageLink(ImportStorageLink):
    storage = models.ForeignKey(YourProviderImportStorage, ...)
    
    class Meta:
        app_label = 'io_storages'  # Required!

Note: This is required for ALL concrete models (not abstract ones) including storage classes and link models.

Django Model Conflict Error

Error: RuntimeError: Conflicting 'providernameimportstorage' models in application 'io_storages'

Cause: Django is registering the same model through two different import paths:

• io_storages.provider.models.ProviderImportStorage (short path)
• label_studio.io_storages.provider.models.ProviderImportStorage (full path)

This happens when:

1. The project directory is in Python's sys.path (which allows io_storages to be imported as a top-level module)
2. Django's URL includes use short module names like 'io_storages.urls'
3. Internal imports mix absolute and relative patterns

Solutions:

Option 1: Use Full Module Paths in URL Includes

# In label_studio/core/urls.py
re_path(r'^', include('label_studio.io_storages.urls')),  # Full path

Option 2: Ensure Consistent Relative Imports All imports within io_storages must use relative imports:

# In models.py, api.py, serializers.py, etc.
from ..base_models import ImportStorage  # Relative
from ..utils import StorageObject       # Relative
from .models import YourProviderStorage  # Relative within provider

Option 3: Avoid Central Model Imports (Temporary) If conflicts persist, temporarily comment out model imports in io_storages/models.py:

# Temporary workaround for import conflicts
# from .yourprovider.models import (
#     YourProviderImportStorage,
#     YourProviderExportStorage,
# )

Critical Requirements:

1. NO absolute imports using from io_storages. within the io_storages package
2. ALL imports within io_storages must be relative (using . or ..)
3. URL includes in Django must use full module paths
4. Models must have explicit app_label = 'io_storages'

This is a complex Django module loading issue that requires careful attention to import patterns.

When in doubt, use this checklist. Proactive implementation following these patterns ensures complete requirements coverage and maintains consistency with existing storage providers.