Generic Rest JSON API

Overview

The Generic Rest JSON API connector is the one you reach for when the system you need has no connector of its own. There is no vendor product behind it. It talks to any HTTP service that accepts a request and returns JSON, which covers most SaaS platforms, most on-premises management consoles with a REST interface, and anything your own teams have built in-house. You supply the domain, the credentials, the endpoint paths and the shape of the response, and ReadyWorks does the transport, the paging, the parsing and the loading.

A connection stores an API domain, a working path on the ReadyWorks server where temporary files are held during processing, and one of 17 authentication methods. Each job under that connection adds a method type, an endpoint path beneath the domain, optional query, header or body parameters, the JSON node that holds the records, and a destination staging table. The response is parsed into rows and columns and written to that table in the ETL5 staging database, where the rest of the platform picks it up.

Every inbound job gets the full retrieval toolkit: five pagination modes (record offset, page number, a node naming the next record, a node naming the next page, or none) and three nested levels of enumeration. Enumeration runs a keys call first, then repeats the data call once per key, substituting the key into the endpoint path as a %token%. Between them, one job can cover an API that insists on a call per tenant, per site or per device, and an API that returns results a hundred at a time.

In ReadyWorks terms this is how a system joins the estate without waiting for a purpose-built connector. Whatever the API returns (hardware inventory, licence assignments, user records, ticket queues, backup coverage) lands in staging alongside everything else and can be reconciled against the assets, people and applications already in the data fabric. That is what turns a standalone tool’s export into something you can plan a migration or a refresh around.

The connector is bidirectional. The outbound side takes a list of fields from ReadyWorks, encodes them as JSON or as form data, sends them to an endpoint with the method you choose, and reads named fields back out of the response so the result of each call is recorded against the record that triggered it. Fields flagged as URI are substituted into the endpoint path rather than the body, which is how you address a specific object in a REST path.

NOTE: Most of the vendor REST connectors in the catalogue are clones of one of these 17 methods. A vendor connector’s original_auth_type names the method it was cloned from, and what that connector adds on top is usually a default domain, a fixed endpoint, a set of job templates and relabelled credential fields. If a vendor connector behaves in a way you did not expect, the recipe described on this page is normally the explanation.

NOTE: Both shipped job templates are skeletons. Neither carries an endpoint, and both are disabled. Clone one, fill in the endpoint, the return data node and the pagination settings, then enable it.

Connector Properties

Authentication Methods

The connector ships 17 authentication methods, all driven by json_rest_api.php. Each one is a recipe in two halves: pre is the authentication call and post is every data call that follows it. Methods holding a static credential place it in the pre header set and make no separate call. Methods that exchange a credential for a token POST to the authentication end point you supply, read the token out of the response using the JSON node you name, and carry it into every data call as %auth_data%.

NOTE: Every method in this connector disables TLS certificate verification. SSL_VERIFYHOST and SSL_VERIFYPEER are both set to false on the authentication call and on the data calls, and the connector definition is locked, so this cannot be changed in the UI. Traffic to an HTTPS endpoint is still encrypted, and the certificate presented by the server is accepted without being checked against a trust store or matched to the host name.

NOTE: Single Authentication defaults to No on all 17 methods. With it set to No, ReadyWorks repeats the authentication call on every pagination and enumeration loop. On the nine methods that perform a token exchange that is an extra token request per page and per enumerated key, which is enough to trip rate limits and to fill the vendor’s audit log with login events. Set it to Yes unless the token expires faster than a job runs.

NOTE: Credential fields (auth_value_1 through auth_value_6, and Password) are stored encrypted. Username, the Text Value fields and the API Domain are stored in clear.

NOTE: API Domain is the scheme and host only. Endpoint paths belong on the job, in API End Point, which accepts %token% substitution so a path can carry an enumerated key, a pagination offset or a credential.

Method 1: No Authentication (JSON Rest API_none)

Connect to a GET method API with no authentication

No credential is sent. Data calls go straight to the endpoint carrying a browser user agent and nothing else. Use it for a genuinely open API, for an internal service reachable only on a trusted network, or for an endpoint that carries its own key in the query string, which you add on the job as an API Data Parameter. It is also the method to choose when the API expects a header shape that none of the other 16 methods produce: leave the connection unauthenticated and add the header per job as an API Data Parameter with the Header box ticked.

Connection Configuration Fields (4)

Authentication Configuration

cURL options:

Method 2: API-Key (JSON Rest API_key)

Connect to a GET method API with an API-Key

Collects a single API Key into the encrypted auth_value_1 field. Be aware that the shipped recipe carries no headers and no body on either the authentication call or the data calls, so selecting this method and pasting a key does not by itself put the key on the wire. Until that is corrected, treat this method as No Authentication plus a place to store the key, and attach the key yourself on each job through API Data Parameters with the Header box ticked, referencing it as %auth_value_1%. If the API expects a conventional header, one of the Bearer Token, x-auth or X-API-KEY methods will be less work.

Connection Configuration Fields (5)

Authentication Configuration

cURL options:

NOTE: The shipped authentication_data for this method contains no headers and no body. Selecting API-Key does not place the key on any request. See the issues section below.

Method 3: Bearer Token (JSON Rest API_token)

Connect to a GET method API with a bearer token

Takes a bearer token you already hold and sends it as Authorization: Bearer <token> with Content-Type: application/json. There is no exchange and no expiry handling, so the token has to be a long-lived one: a personal access token, a service account token, or an API key the vendor happens to accept in the Authorization header. This is the right choice for the large number of platforms that issue a static token from an admin console and expect it back verbatim. If the vendor’s token expires on a short cycle, pick one of the POST methods that fetches a fresh token instead.

Connection Configuration Fields (5)

Authentication Configuration

Headers sent on the authentication call:

cURL options:

Method 4: Header: x-auth (JSON Rest API_xauth)

Connect to a GET method API with an x-auth token

Identical in shape to Bearer Token, sending the stored value as a bare x-auth: header with Content-Type: application/json. Use it for APIs that define their own authentication header rather than reusing the standard Authorization header, which is common on appliance and storage management interfaces. The value is stored encrypted in auth_value_1.

Connection Configuration Fields (5)

Authentication Configuration

Headers sent on the authentication call:

cURL options:

Method 5: Header: X-API-KEY (JSON Rest API_xapikey)

Connect to a GET method API with a x-api-key token

Sends the stored value as an X-API-KEY: header with Content-Type: application/json. Functionally this is the x-auth method under a different header name, and you pick between them purely on what the vendor’s documentation says the header is called. One difference is worth knowing: this method stores the key in the user_pass column rather than in auth_value_1, so the value does not carry across if you later switch the connection to a different method.

Connection Configuration Fields (5)

Authentication Configuration

Headers sent on the authentication call:

cURL options:

NOTE: The key is stored in the user_pass column, unlike the other key-bearing methods, which use auth_value_1. Switching an existing connection to or from this method means re-entering the key.

Method 6: Custom Header Fields (JSON Rest API_custom)

Connect to a GET method API with custom header fields

Lets you define an arbitrary list of header name and value pairs on the connection, which are then sent on every call. It was the escape hatch for APIs whose header requirements none of the fixed methods matched. It is disabled in the shipped catalogue and the underlying Custom Header Fields template is marked deprecated in favour of the authentication fields. Use No Authentication with per-job API Data Parameters marked as headers to achieve the same result.

Connection Configuration Fields (4)

Authentication Configuration

cURL options:

NOTE: This method is disabled in the shipped catalogue and does not appear in the connection form.

Method 7: Basic Authorization: Username / Password (JSON Rest API_user)

Connect to a GET method API with a username / password

Standard HTTP Basic authentication. Username and Password are both required, and ReadyWorks assembles username:password, Base64-encodes it and sends it as Authorization: Basic. There is no separate authentication call, so the credential goes on every data request. Pick this when the vendor documents Basic auth and expects the credential on each call, which is still the norm for appliance APIs and for older on-premises consoles. Note that the password is sent on every request, so the Single Authentication setting makes no difference to the number of credential exposures here.

Connection Configuration Fields (6)

Authentication Configuration

Headers sent on the authentication call:

cURL options:

Method 8: POST username / password – return session cookie (JSON Rest API_user_sess)

Connect to a POST method API with a user/ password retrieving value for session cookie request header

A session-based exchange. ReadyWorks POSTs to the Session ID End Point with the username and password Base64-encoded in an Authorization: Basic header, reads the returned session identifier out of the JSON node you name in Session ID JSON Node, and sends it on every subsequent data call as a Cookie: header. Choose it for APIs that hand out a session on login and then expect that session back as a cookie, a pattern common in virtualisation and storage management consoles. You must fill in both the end point and the JSON node; neither is marked required on the form, and the method cannot work without them.

Connection Configuration Fields (8)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

cURL options:

NOTE: Session ID End Point and Session ID JSON Node are both optional on the form and both mandatory in practice.

Method 9: Get/Set Session Cookie from API Token (JSON Rest API_token_cook)

Use an API Token to get, and then set, a session value

For APIs that trade a long-lived API token for a short-lived session cookie. The authentication call carries the >SetCookie directive, which tells the driver to capture whatever Set-Cookie headers the response returns into a cookie jar, and every data call carries >GetCookie, which replays them. The API Token itself is not placed in a header by the recipe, so put it in the end point: the API Token Get Session End Point field accepts %token% substitution, which lets you write something like /auth/session?token=%auth_value_1%. Use this when the vendor’s login call answers with a Set-Cookie rather than with a token in the response body.

Connection Configuration Fields (6)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

cURL options:

NOTE: The API Token Get Session End Point field accepts %token% substitution, which is how the API Token reaches the authentication call.

Method 10: POST username / password – return bearer token (JSON Rest API_user_token)

Connect to a POST method API with a username / password retrieving value for bearer token

POSTs to the Bearer Token End Point with the username and password Base64-encoded in an Authorization: Basic header, extracts the token from the JSON node named in Bearer Token JSON Node, and sends it on every data call as Authorization: Bearer. This is the most common shape among modern enterprise APIs: log in with a credential, get a short-lived bearer token, use the token. Pick it over the plain Bearer Token method whenever the token expires, because ReadyWorks fetches a fresh one each run. Both the end point and the JSON node must be filled in for the method to work.

Connection Configuration Fields (8)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

cURL options:

NOTE: Bearer Token End Point and Bearer Token JSON Node are both optional on the form and both mandatory in practice.

Method 11: POST username / password – return authorization token (JSON Rest API_user_auth)

Connect to a POST method API with a username / password retrieving value for authorization token

POSTs a JSON body of {"username": ..., "password": ...} to the Authentication Token End Point, extracts the token from the node you name, and sends it on the data calls in a bare Authorization: header with no Bearer prefix. The distinction from the bearer variant is entirely in those two details: the credential travels in the request body rather than in a Basic header, and the token is sent unprefixed. Match this against what the vendor’s documentation shows in its example request, because an API that wants Authorization: Bearer <token> will reject the value this method sends.

Connection Configuration Fields (8)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

Authentication request body:

cURL options:

Method 12: POST key / secret – return bearer token (JSON Rest API_key_token)

Connect to a POST method API with a key/ secret retrieving value for bearer token

Machine-to-machine authentication with a Client Id and Client Secret. Both are POSTed as form-urlencoded fields named client_id and client_secret to the Auth-Token End Point, and the token read from the response is sent as Authorization: Bearer. Use it where the vendor issues an application credential pair rather than a user login, which is the usual arrangement for integrations that must survive staff changes. The request body carries only the two credential fields, so an OAuth 2.0 token endpoint that insists on grant_type or scope will reject it; for Microsoft Entra, use the MS Graph method instead.

Connection Configuration Fields (8)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

Authentication request body:

cURL options:

Method 13: POST key / secret – return auth-token (JSON Rest API_key_auth)

Connect to a POST method API with a key/ secret retrieving value for auth-token

The same credential pair under different names. An API Key and a Secret Key are POSTed as a JSON body with the keys apiKey and secretKey, and the token returned is sent on data calls in an auth-token: header. Choose between this and the key/secret bearer method by looking at two things in the vendor’s documentation: whether the login body field names are apiKey/secretKey or client_id/client_secret, and whether the resulting token is presented as auth-token or as Authorization: Bearer.

Connection Configuration Fields (8)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

Authentication request body:

cURL options:

Method 14: MS Graph Credentials GET Bearer Token (JSON Rest API_ms_graph)

Connect to a MS Graph method API with application credentials retrieving GET bearer token for data calls.

OAuth 2.0 client credentials against Microsoft Entra ID for calling Microsoft Graph. You supply the directory (tenant) Id, the application (client) Id and a client secret from an Entra app registration. ReadyWorks POSTs client_id, client_secret, grant_type=client_credentials and scope=https://graph.microsoft.com/.default as form data to the token endpoint, which defaults to https://login.microsoftonline.com/%auth_value_1%/oauth2/v2.0/token, reads access_token out of the reply and sends it as Authorization: Bearer. Everything the app registration has been granted application permissions for is then reachable: Intune managed devices, Entra users and groups, licence assignments, Autopilot registrations. The grant type and the scope are fixed in the locked connector definition, so this method issues Graph tokens only.

Connection Configuration Fields (9)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

Authentication request body:

cURL options:

NOTE: The scope is fixed at https://graph.microsoft.com/.default inside the locked connector definition. This method cannot be repurposed to obtain a token for a different Entra-protected API.

Method 15: GSSAPI (JSON Rest API_gssapi)

Connect to a GET method API with GSSAPI

Kerberos, through libcurl’s SPNEGO negotiation (HTTPAUTH set to the negotiate value, with unrestricted authentication and a cookie jar enabled on both the authentication and the data calls). The ReadyWorks host must already hold a valid Kerberos ticket for the service principal, because the method uses the host’s credential cache rather than sending a password. Use it for intranet APIs published behind Windows Integrated Authentication or behind a Kerberised reverse proxy, where a username and password would not be accepted. The Password, end point and header node fields appear on the form but are not referenced by the recipe.

Connection Configuration Fields (8)

Authentication Configuration

Headers sent on the authentication call:

cURL options:

NOTE: The connection form offers Password, Username / Password API End Point and Username / Password Request Header Node. None of the three is referenced by the authentication recipe.

Method 16: POST username / password / domain – return bearer token (JSON Rest API_cohesity)

Authentication used for Cohesity (Storage Solution)

A three-part credential login, built originally for Cohesity. Username, Password and Domain are POSTed as a JSON body to the Bearer Token End Point, which defaults to /irisservices/api/v1/public/accessTokens, and the value at the accessToken node is sent on the data calls as Authorization: Bearer. The domain field is what distinguishes it: use this method for any API whose login body needs a third field alongside the username and password, typically an Active Directory domain or a tenant name. Change the end point and the JSON node defaults if you are pointing it at something other than Cohesity.

Connection Configuration Fields (9)

Authentication Configuration

Headers sent on the authentication call:

Headers sent on the data calls:

Authentication request body:

cURL options:

Method 17: Digest Authentication with username / password (JSON Rest API_jetpatch)

Used for JetPatch Digest Authentication API

HTTP Digest authentication, handled by libcurl (HTTPAUTH set to CURLAUTH_DIGEST with the username and password in USERPWD) on both the authentication call and the data calls. Digest challenges the client with a nonce and receives a hash rather than the password itself, so the password never crosses the wire in a reusable form. Named for JetPatch, it works for any API that answers with a WWW-Authenticate: Digest challenge. Pick it when Basic authentication returns 401 and the response header names Digest.

Connection Configuration Fields (6)

Authentication Configuration

Headers sent on the authentication call:

cURL options:

Inbound Job Fields Enabled (44)

Inbound jobs expose 44 fields, effectively the entire set the ETL5 engine offers for an HTTP source: request method and endpoint, parameters and raw body, the response node to parse, five pagination modes, three nested levels of enumeration, and retry behaviour.

NOTE: API End Point, API Data Parameters, Raw JSON Body Data, API Return Data Node, API Pagination Node, Pagination Termination Node, the enumeration paths, nodes and field lists and Additional Fields all accept %token% substitution at run time. That is what lets one job walk a collection or iterate a set of keys.

NOTE: Retry Pause takes three values: how many times to retry a failed call, how many seconds to wait between attempts, and the API timeout in seconds. Set it on any job pointing at an API that rate-limits or occasionally stalls, because the default is no retry.

NOTE: Pause Enumeration takes two values, a call count and a number of seconds, and inserts a wait every so many calls. Use it together with Kill Enumeration on Error when enumerating over more than a few hundred keys.

NOTE: Several job settings carried on the job record are not exposed on this connector’s form, including Allow Empty Table, Convert UUID-Keyed Objects to Rows, Append Files to Same Destination Table and Ignore XML Attributes.

Outbound Job Fields Enabled (13)

Outbound jobs expose 13 fields: the request method and endpoint, how the payload is encoded, the list of fields sent, the response fields read back into ReadyWorks, and an optional raw JSON template that replaces the generated body entirely.

NOTE: In Outbound Fields List, a field with the URI box ticked is substituted into the endpoint path as %<name>% rather than added to the body. That is how you address a specific object, for example PATCH /devices/%device_id%.

NOTE: Method Type defaults to GET on a new outbound job. Most outbound work needs POST, PUT or PATCH, so set it deliberately.

NOTE: Return Data Fields List uses dotted paths to reach into the response, so a nested value is addressed as data.record_key. The values it names are written back against the ReadyWorks record that triggered the call, which is what gives you a per-record success or failure trail.

Inbound Job Templates (1)

One inbound template ships. It is a skeleton for you to clone rather than a job that does anything on its own, and it is disabled.

Job Template Configuration

Settings

NOTE: Clone the template rather than editing it, and give each job its own destination table name. Leaving several jobs pointed at api_data means they overwrite one another.

Outbound Job Templates (1)

One outbound template ships, and like the inbound one it is a disabled skeleton.

Job Template Configuration

Settings

Job Parameters and Enumeration

Rest POST API

Outbound fields list:

Return data fields:

NOTE: The template sends JSON. Switch Data Sending Method to Form Data for APIs that expect application/x-www-form-urlencoded, and use Raw JSON Outbound Template when the API needs a nested body structure that the flat outbound field list cannot express.