Data and Information Feed
This section provides general information about data and information feeds provisioned by and/or via DECAF.
Introduction
DECAF provides various APIs which expose both core and auxiliary functionalities of the platform. In addition to these APIs, we provide file-based data and information feeds over SFTP and HTTPS protocols.
Structure of layout, naming convention, data format and content of these files are specific to individual use cases and applications. Therefore, we are going to focus on the general aspects of the data and information feed in this guide.
Application specific documentation is provided on a case-by-case basis.
Protocols
There are two possible ways of using the data and information feed:
- SSH File Transfer Protocol (SFTP) for read and write operations (depending on the application)
- Hypertext Transfer Protocol Secure (HTTPS) for read operations.
The main implemetation is SFTP. HTTPS protocol is an auxiliary means to access the same data that is available over SFTP.
The default mode of operation is SFTP Pull, ie. feed consumers will pull data from our SFTP servers. We can also comply with SFTP Push mode, ie. if the feed consumer has its own SFTP server, we can push the data to the consumer using the SFTP service account provisioned.
Security
SFTP option allow us to whitelist/blacklist IP addresses and use public key authentication or password authentication mechanisms. HTTPS option, on the other hand, only provides HTTP Basic Authentication mechanism.
In both cases, files can be encrypted via PGP (OpenPGP) if the feed consumer decides to opt-in for this alternative.
Accessing the Service
Accessing SFTP Service
Before we provision the feed to the consumer, we will ask for:
| Name | Notes |
|---|---|
| SSH Public Key | Applicable only if the consumer opts-in to public key authentication method. |
| PGP Public Key | Applicable only if the consumer opts-in to file encryption. |
| IP Addresses | Applicable only if the consumer opts-in to IP whitelisting. |
Once we provision the feed to the consumer, we will provide following information:
| Name | Example | Notes |
|---|---|---|
| Hostname | example.decafhub.com | Hostname of the SFTP service. |
| Username | my-username | Username credential of the SFTP service. |
| Password | my-password | Password credential of the SFTP service. This applies if the consumer opts-in to password-based authentication. |
Accessing HTTPS Service
If the consumer requests HTTPS service in addition to SFTP service, we will provide following information:
| Name | Example | Notes |
|---|---|---|
| URL | https://example.decafhub.com | Base URL of the HTTPS feed service. |
| Username | my-username | Username credential of the HTTPS feed service. |
| Password | my-password | Password credential of the HTTPS feed service. |
Communication of Credentials
Once the feed is setup, we communicate credentials over a secure channel which allows us to restrict the recipients to selected people. In particular, recipients of credentials will receive an email containing a secure link to a Web page which will prompt the recipients email address. Once the recipient provides the email address, a one-time token will be send to the email address. This one-time token will be prompted on the same Web page. Upon successful one-time token submission, credentials will be shown on the Web page.
Example Usage over SFTP
Let's assume that the SFTP service is hosted on example.decafhub.com. You can
use a GUI-based SFTP client (such as WinSCP or Cyberduck -- not an endorsement).
A typical console based session is as follows:
$ sftp redacted-username@example.decafhub.com
redacted-username@example.decafhub.com's password:
Connected to example.decafhub.com.
sftp> pwd
Remote working directory: /
sftp> ls -la
drwxr-xr-x 1 0 0 0 Sep 16 04:16 .
drwxr-xr-x 1 0 0 0 Sep 16 04:16 outgoing
sftp> cd outgoing
sftp> ls -la
drwxr-xr-x 1 0 0 0 Sep 16 04:16 .
drwxr-xr-x 1 0 0 0 Sep 16 04:16 ..
-rw-r--r-- 1 1000 100 93 Sep 14 07:09 README.md
-rw-r--r-- 1 1000 100 766 Sep 16 04:13 positions_2022-09-14.csv
-rw-r--r-- 1 1000 100 1129 Sep 16 04:13 positions_2022-09-14.json
sftp> get positions_2022-09-14.csv
Fetching /outgoing/positions_2022-09-14.csv to positions_2022-09-14.csv
/outgoing/positions_2022-09-14.csv
sftp> exit
Example Usage over HTTPS
For example, the URL schema for the above files is as follows:
https://example.decafhub.com/api/files/positions_<ISO-8601-FORMATTED-DATE>.[csv|json]
Note that the URL schema is dependent on the use case or application itself.
The service uses HTTP Basic Authentication Schema (RFC7617).
A typical HTTPS session reads as follows (using curl command):
$ curl -D - -u <redacted-username> https://example.decafhub.com/api/files/positions_2022-09-14.json
Enter host password for user '<redacted-username>':
HTTP/2 200
accept-ranges: bytes
content-type: application/json
last-modified: Fri, 16 Sep 2022 04:13:12 GMT
content-length: 1129
date: Fri, 16 Sep 2022 04:26:37 GMT
<redacted JSON payload>
Retention Policy
Until further notice and unless specified otherwise (based on application), files are retained indefinitely. This policy may change in the future.
Support Request
Please feel free to submit your support requests, questions and feedback to support@telostat.com.