Use the Data Streamer

You can manage data streams with the emnify Portal or the Data Streamer API.

Choose which interface to use

emnify Portal
Managing data streams with the Portal is convenient when taking the first steps or if you don't expect configuration changes to be needed often.

Data Streamer API
Integrating the Data Streamer API becomes a faster and more secure approach when multiple data streams need to be managed as part of the daily business or in the case of advanced operations (for example, seamless migration from one data stream to another without lags or duplicates).

Data Streamer in the Portal

To manage your data streams, log in to your emnify Portal account. Then, navigate to the Data Streams section of the Integrations page.

View data streams

If there are no data streams configured, the Data Streams panel displays all available connection types as tiles.

If you already have a data stream configured, the panel displays all existing streams, ordered by creation date (newest on the top). This list view can be used to gain an overview of the current states of different data streams.

Data streams are marked as Running are properly operating. Failed streams exhibit an Error status. Paused streams are marked as Paused.

The Details button allows you to inspect the configuration of the specific stream. The Destination configuration varies depending on the connection type.

Create data streams

When no configured data streams are available, click Add on the preferred connection type tile. Otherwise, click Add New Stream at the top of the existing Data Streams list. This shows the connection type tiles. You can choose your preferred connection type and click Add.

Configure the source by selecting your desired stream type. Optionally, you can add event options for filtering on event types.

Provide the required settings to configure the destination.

info

API keys and configuration parameters differ by connection type. Instructions for configuring each connection type are in the Available integrations section.

Click Create and verify the status of your new stream.

If everything goes well, the data stream is successfully created. The configuration view closes, and you see the integration status shown as Running.

warning

If the configuration is invalid, a warning displays along with an error message indicating the cause of the problem. If this happens, the stream isn't created and you must correct the configuration.

Inspect and update data streams

With the Details button on the Data Stream tiles, you can inspect the configuration and update the filters for event data streams.

Delete data streams

To permanently delete a data stream, click the trash bin icon near the Details button.

note

You'll need to confirm this action.

Data Streamer API

You can manage your data streams using the emnify Data Streamer API. Available entry points are listed under the Integrations section of the emnify REST API reference.

info

You can find detailed information on the Data Streamer data properties and compatibility notes in the Data Streamer API Reference in the emnify System Documentation.

Authentication

To use the emnify REST API, you need to authenticate with an authentication token. emnify uses JWTs as the authentication token.

Each integration also requires credentials to verify that the Data Streamer has permission to write data to that service.

tip

The Getting Started guide in the emnify REST API Documentation provides step-by-step instructions for retrieving this token.

Specification and documentation

The emnify REST API is based on the OpenAPI Specification OAS3. There's also interactive API documentation.

tip

Prior OpenAPI knowledge isn't necessary for working with the emnify API. Nevertheless, understanding how an OpenAPI Specification is structured could help you navigate the documentation.

The Data Streamer reference page in the emnify REST API Documentation provides details about the data structure and common elements you'll encounter while working with the Data Streamer API.

Work with the Data Streamer API

info

API keys and configuration parameters differ by connection type. Details on setting up, updating, or deleting integrations are in the emnify API reference. It lists all entry points for managing data streams programmatically and contains examples of the available connection types and filtering options.

The following entry points are available for managing data streams:

Method	Entrypoint	Description
GET	/api/v2/data_stream	List data streams.
GET	/api/v2/data_stream/{data_stream_id}	Retrieve detailed information of a data stream by ID.
POST	/api/v2/data_stream	Create a data stream.
PATCH	/api/v2/data_stream/{data_stream_id}	Update a data stream by ID.
POST	/api/v2/data_stream/{data_stream_id}/restart	Restart a data stream by ID.
DELETE	/api/v2/data_stream/{data_stream_id}	Delete a data stream by ID.

The following lookup entry points are available for retrieving possible configuration options:

Method	Entrypoint	Description
GET	/api/v2/data_stream/status	List possible data stream statuses.
GET	/api/v2/data_stream/connection_type	List possible data stream connection types.
GET	/api/v2/data_stream/type	List possible data stream types.
GET	/api/v2/data_stream/filter_field_type	List possible data stream filter fields.

Create data streams

Once authenticated, users can create data streams by sending POST requests to /api/v2/data_stream.

An example cURL to create a data stream would look like the following:

curl -X POST "https://cdn.emnify.net/api/v2/data_stream" \
-H "accept: */*" \
-H "Authorization: Bearer AuthToken" \
-H "Content-Type: application/json" \
-d '{"data_stream_type":...'

The request body (set using the -d flag in cURL) configures the data stream's parameters.

The following JSON request body example shows how to create a stream of usage data records with the Webhook integration:

{
  "data_stream_type": {
    "id": 1 // usage data
  },
  "destination": {
    "connection_type": "RestAPI",
    "credentials": {
      "url": "https://my-application-server/servicebus.net",
      "method": "POST", // HTTP method expected by the application server for pushing data records
      "headers": [
        // can be used for authentication or
        // other mapping functionality on the application server side
        "my_custom_header_1:value1",
        "my_custom_header_2:value2"
      ]
    },
    "format": "Json" // further format options will be added in future
  }
}

You can also create an event stream with AWS Kinesis integration and enabled filtering, like the following JSON request body example:

{
  "data_stream_type": {
    "id": 2 // events
  },
  "destination": {
    "connection_type": "AwsKinesis",
    "credentials": {
      "region": "af-south-1",
      "stream_name": "demo-stream",
      "role_arn": "arn:aws:iam:1234567890:role/role_for_kinesis_data_stream"
    },
    "format": "Json"
  },
  "filters": [
    {
      "field": "event_type_id",
      "value": [1, 2, 3]
    },
    {
      "or": [
        {
          "field": "alert",
          "value": true
        }
      ]
    }
  ]
}

API reference

Create data stream.

List data streams

You can list all data streams belonging to your organization by sending GET requests to /api/v2/data_stream.

An example cURL request to list data streams would look like the following:

curl -X GET "https://cdn.emnify.net/api/v2/data_stream" \
-H "accept: application/json"  \
-H "Authorization:Bearer AuthToken"

API reference

List data stream configurations of your organization

Retrieve details

Retrieving details of an existing data stream is possible by making a GET request to /api/v2/data_stream/{data_stream_id}.

The data_stream_id path parameter is the top-level id property in each object returned by the list data streams request and is an ID that's unique for each data stream.

An example cURL request to get detailed information for a data stream with an ID of 123 would look like the following:

curl -X GET "https://cdn.emnify.net/api/v2/data_stream/123" \
-H "accept: application/json" \
-H "Authorization:Bearer AuthToken"

API reference

Get details on existing data stream

Update data streams

Authenticated users may update data streams belonging to their organization by making a PATCH request to /api/v2/data_stream/{data_stream_id}.

The data_stream_id path parameter is the top-level id property in each object returned by the list data streams request and is an ID that's unique for each data stream.

An example cURL request to update a data stream with an ID of 123:

curl -X PATCH "https://cdn.emnify.net/api/v2/data_stream/123" \
-H "accept: application/json" \
-H"Authorization: Bearer AuthToken" \
-H "Content-Type: application/json" \-d '{...}'

The request body (set using the -d flag in cURL) configures the parameters of the data stream itself.

The following example shows how to pause a stream and erase any filters:

{
  "status": {
    "id": 2 // 1 = Running, 2 = Paused, 3 = Error, 4 = Pending
  },
  "filters": []
}

API reference

Modify existing data stream

Delete data streams

Sending a DELETE request to /api/v2/data_stream/{data_stream_id} deletes the data stream with that ID.

The data_stream_id path parameter is the top-level id property in each object returned by the list data streams request and is an ID that's unique for each data stream.

The following cURL request would then delete a data stream with an ID of 123:

curl -X DELETE "https://cdn.emnify.net/api/v2/data_stream/123" \
-H "accept: application/json" \
-H "Authorization: Bearer AuthToken"

API reference

Delete existing data stream