Data Extract API

Introduction

The Data Extract API provides built-in endpoints for programmatically transferring your solution data into data warehouses and data lakes for further analysis and storage. This API is designed to facilitate seamless integration of your Appfarm solution data into existing data transformation workflows.

Common data warehouse software that you can use with this API includes:

• Snowflake

• Databricks

• Google BigQuery

• Amazon Redshift

As a developer, you're responsible for creating and managing your own processes for calling the endpoints and extracting your data. This is typically managed through ETL (extract, transform, load) or ELT (extract, load, transform) software.

Key features

1. Object class-level export: Data export can be enabled per object class, allowing for targeted data extraction. A unique endpoint is generated for each object class.

2. Efficient extraction: The Data Extract API endpoints stream data and use cursor-based pagination, enabling efficient extraction of large datasets.

3. Secure access: Calling a Data Extract API endpoint requires an API key generated within a service account with explicit data export permissions.

Usage

• Requests must comply with the solution's allocated number of inbound HTTP requests as per the Product Glossary and Acceptable Use Policy.

Configuration

Enabling the Data Extract API

Premium feature

The Data Extract API is a premium feature. Availability is determined by your subscription. Please check your current subscription or contact customer success to ensure you have access to this functionality.

To enable the Data Extract API for a specific object class:

1. In the object class properties, check the Enable Endpoint box to enable data export for that specific object class.

2. Hover over the blue dot next to Enable Endpoint and note the object class' unique endpoint.

The Data Extract API is available in all environments when enabled in a solution.

Supported data

• The API primarily supports exporting data from standard object classes.

• File object classes are supported, but only metadata is exported (file content URLs are not included).

• Time series data is not supported by this API and requires separate APIs for access.

Pagination

The Data Extract API uses cursor-based pagination:

• Each response includes a next_cursor for fetching the next page of objects.

• By default, 1,000 objects are returned per response.

• The limit query parameter can adjust the number of returned objects (max 50,000).

Making requests

Endpoint structure

Each object class has a unique endpoint for data extraction. The subdomain in the endpoint URL is specific to your deployment environment (e.g., Development, Test, Production).

For Production, the general structure of the endpoint is as follows:

GET https://{SOLUTION_SHORTNAME}.appfarm.app/api/data_export/{OBJECT_CLASS_ID}

Example of an object class endpoint across environments:

// Development
https://thirty50-dev.appfarm.app/api/data_export/SFjOV9

// Test
https://thirty50-test.appfarm.app/api/data_export/SFjOV9

// Staging
https://thirty50-stage.appfarm.app/api/data_export/SFjOV9

// Production
https://thirty50.appfarm.app/api/data_export/SFjOV9

Authentication

Requests to the Data Extract API require a Bearer token for authentication. To set up authentication:

1. Use an existing service account or create a new one in Appfarm Create.

2. Ensure the service account has a role with Read permissions for the object class you want to access.

3. Generate an API key for the service account. Make sure the API key has the Data Export scope.

4. Use this API key as a Bearer token in your requests for authentication.

For more details on service accounts and API keys, refer to the Service accounts documentation.

Query parameters

• cursor (optional): The ID of an object to determine the starting point for the next set of results.

• limit (optional): The maximum number of objects to return (default: 1,000, max: 50,000).

Example request with query parameters:

GET https://thirty50.appfarm.app/api/data_export/SFjOV9?cursor=64e8a2495b240f565557514c&limit=2000

Response format

The API returns a JSON object containing:

• data: An array of objects (e.g., transactions).

• next_cursor: The ID of the next object, or null if there are no more objects.

Example response:

{
    "data": [
        {
            "_id": "636053da31099daf97cf90bd",
            "startDate": "2022-12-01T23:00:00.000Z",
            "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
            "image": "https://images.unsplash.com/photo-1531058020387-3be344556be6?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=Mnw5MDg2M3wwfDF8c2VhcmNofDE2fHxldmVudHxlbnwwfHx8fDE2NjcyNTU3MDk&ixlib=rb-4.0.3&q=80&w=1080",
            "title": "Lorem ipsum dolor sit amet",
            "type": "63604fb180739f1a3363c94e",
            "openForRegistration": true,
            "af_randomId": "qirp2j9Raei2Z8mrVv93U",
            "af_createdDate": "2022-10-31T23:01:46.217Z",
            "af_updatedDate": "2022-11-15T14:05:20.634Z",
            "af_updatedBy": "62ecd53ad4077713c99cb873"
            "__v": 0,
        },
        // Additional objects...
    ],
    "next_cursor": "64e8a2495b240f565557514c"
}