---
name: eazybi-custom-sources
description: >
  Build eazyBI custom source apps — JavaScript packages that add a new source application
  to eazyBI: application.js manifest (source params, authentication, import options), schema_definition.js
  OLAP dimensions and cubes, api.js REST or GraphQL requests and SQL queries, importer.js and importers/*.js writing rows
  into the eazyBI data warehouse, sample_reports.js reports and dashboards, and SVG icons. Use when
  creating, reviewing, debugging or extending an eazyBI custom source app, or when asked to
  import data from an external REST or GraphQL API or from a SQL database into eazyBI as a new source
  application.
metadata:
  author: eazybi
  version: "1.0"
---

# eazyBI custom source apps

A custom source app is a directory of JavaScript files that eazyBI installs and runs as a
first class source application. Once installed, a user adds it as a source application to an account and
gets its own icon, connection form, source selection, OLAP cube with dimensions and measures, importer
with incremental import support, and sample reports — without any eazyBI server side code. An app reads its
data from a REST or GraphQL API, from a SQL database, or from both.

The key declared in the manifest is prefixed with `custom_`: manifest `key: "github"` becomes the
source application type `custom_github` and its data warehouse tables are named `x_github_*`.

## Implementation directory

```
github/                     directory name is free, the application type comes from application.js
├── application.js          required — manifest: name, source params, authentication, import options
├── schema_definition.js    required — dimensions, cubes, measures (the OLAP data model)
├── importer.js             required — app.importer.importAll(), the import entry point
├── api.js                  optional — all requests to the source system
├── importers/
│   ├── repositories.js     optional — per object importers, one file per imported object
│   └── pull_requests.js
├── sample_reports.js       optional — sample reports and dashboards created after the first import
├── icon.svg                optional — square source application icon
├── icon_dark.svg           optional — dark theme icon (defaults to icon.svg)
├── test/
│   ├── api_test.js         optional — tests, run on the server against mocked responses
│   ├── importer_test.js
│   ├── importers/
│   │   └── repositories_test.js
│   └── fixtures/
│       └── repositories.json
├── README.md               optional — documentation for the users of the app
├── CHANGELOG.md            optional — notable changes of every published version
└── LICENSE                 optional — license text (with license: "MIT" in the manifest)
```

Only these files are allowed in a package: the JavaScript files above (`importers/*.js` file names must be
lowercase `[a-z][a-z0-9_]*.js`), the test files (`test/*_test.js` and `test/importers/*_test.js`), the two
icon file names, `README`, `CHANGELOG` and `LICENSE` files (also with the `.md` or `.txt` extension), and `*.json`
files. Other markdown files (`AGENTS.md`, `CLAUDE.md`, development notes) are ignored and not installed.
Limits: at most 200 files, 2 MB per file, 10 MB in total.

Write a `README.md` for the users of the app and keep a `CHANGELOG.md` next to it in the
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format: an `Unreleased` section for the changes since
the last published version, and one section per published manifest `version` with its date and its changes
grouped under `Added`, `Changed`, `Fixed` and `Removed`. Number the versions by
[Semantic Versioning](https://semver.org/spec/v2.0.0.html) — when to increment the version and how to pick
the number is described with the `version` option in
[references/application-manifest.md](references/application-manifest.md).

Only JavaScript files are loaded — there is no way to read a file at runtime, so the bundled `README.md` and
`*.json` files are documentation and metadata for people and tooling, not data for the implementation. Static data
(field mappings, lookup tables) belongs in an `importers/*.js` file as a data namespace:

```js
// importers/field_mapping.js — app.importers.fieldMapping, available to importer.js and importers/*.js
app.importers.fieldMapping = {
  columnsByFieldType: {string: "text_value", number: "numeric_value"}
};
```

### Repository layout

A git repository contains either one custom source app or several:

- **one app** — put `application.js` and all the other implementation files in the top directory of the
  repository, and install it with `eazybi custom-source-apps install https://github.com/owner/repository`
  (or from Bitbucket Cloud with `https://bitbucket.org/workspace/repository`);
- **several apps** — give each app its own subdirectory, by convention named after the
  application type (`github/`, `gitlab/`), and install each one with its own `--subdir github`.

A private repository is installable when an eazyBI administrator has configured its credentials in the site
advanced settings (`[source_application.custom.github_tokens]` with a personal access token, or
`[source_application.custom.bitbucket_credentials]` with the Atlassian account email and an API token,
referencing a site secret) or in the instance settings; the CLI and the install form take no credentials.

Add an `AGENTS.md` file to the root directory of the repository, so that every agent session reads this
skill before it changes anything:

```md
# Custom source apps

Read the eazyBI custom source apps skill at
https://eazybi.com/skills/eazybi-custom-sources and follow it in this repository.
```

## Files and namespaces

Every implementation file assigns its own namespace under the global `app` object, derived from its file
name — a missing or misspelled namespace fails at install time:

| File | Namespace | Defines |
|---|---|---|
| `application.js` | — | calls `app.defineApplication({...})` |
| `schema_definition.js` | — | calls `app.defineSchema((schema) => {...})` |
| `sample_reports.js` | — | calls `app.defineSampleReports((samples) => {...})` |
| `api.js` | `app.api` | `testConnection`, `getSourceSelections`, `getAvailableCustomFields`, own request functions |
| `importer.js` | `app.importer` | `importAll()` — invoked by eazyBI to run the import |
| `importers/pull_requests.js` | `app.importers.pullRequests` | per object import functions |

File names are snake_case, namespaces are camelCase. Which files are loaded together depends on what eazyBI
is doing, so each group must be self-contained:

- `application.js`, `schema_definition.js` and `sample_reports.js` are each evaluated alone with their own
  definition function — they can not use anything defined in the other files. `schema_definition.js` may
  read the import options chosen by the user (`source.getSourceSelection()`, `source.getExtraOption(key)`,
  `source.getCustomFields()`) and the entered source params (`source.getSourceParam(key)`), so that the
  schema can depend on them.
- `api.js` is evaluated alone when the connection is tested and when the source selection or the custom
  fields list is built, so `testConnection`, `getSourceSelections` and `getAvailableCustomFields` can only
  use `app.api` functions of the same file.
- The import loads `api.js`, `importer.js` and all `importers/*.js` together, so importers can call
  `app.api` functions and each other.

Every file is evaluated wrapped in its own function scope, so its top level `var`, `let`, `const` and
`function` declarations are private to that file — two files may declare a constant or a helper function of
the same name without overwriting each other. Anything that another file needs must be assigned to an `app`
namespace (for example `app.api.STATE_EVENT_TYPES`) and addressed by its full path.

Three global namespaces are available at runtime:

- `app` — this implementation (its own functions),
- `source` — the source system and the current import session (requests, import options, source params,
  progress); in `schema_definition.js` only the import option and source param getters are available,
- `dwh` — the eazyBI data warehouse tables derived from `schema_definition.js`.

Define namespace members with the method syntax and call siblings of the same namespace through `this`;
address other namespaces by their full path (`app.api.getCommits`). Namespace functions are always invoked
as methods of their own namespace, also the ones eazyBI calls (`importAll`, `importOne`, `testConnection`,
`getSourceSelections`). Arrow functions do not bind `this` — use them for callbacks (where they conveniently
inherit the `this` of the enclosing method), never for namespace members.

```js
app.importers.repositories = {
  importOne(repository) {
    var rows = app.api.getCommits(repository.full_name).map((commit) => this.buildCommitRow(repository, commit));
    dwh.measures.commits.importRows(rows);
  },

  buildCommitRow(repository, commit) { /* ... */ }
};
```

## JavaScript engine

The files run in a sandboxed Rhino engine with ES6 language level. Available: arrow functions, `const` /
`let`, template literals, object literal method syntax, destructuring, default parameters, spread,
`for…of`, optional chaining, `JSON`, `Math`, `Date`, `RegExp`.

Restrictions:

- No ES6 `class` declarations — use object literals for namespaces, plain functions for constructors.
- No rest parameters in arrow functions.
- No `async` / `await` — all code is evaluated synchronously, including every `source.*` request.
- `const` is not correctly block scoped per loop iteration: in `for` loops use `let` for the iteration
  variable and for variables declared in the loop body (`for (let item of items)`, `let value = ...`).
- No `require`, no module loading, no file system, no timers, no direct network access — the only way out
  is the `source` namespace.
- File top level code must be side effect free: it may only define namespaces and constants. Host functions
  are not available while the files are validated at install time, so calling one at the top level fails
  the install.

Additional globals available in every file:

- `logger.info(...)`, `logger.warn(...)`, `logger.error(...)` and `logger.debug(...)` (dropped unless the
  `logger.level` advanced setting is `"debug"`) — written to the import log, and
  `logger.startImport(name, objectName)` / `logger.finishImport(name, importedCount)` during an import to
  log each imported object and the profiling results of each importer (see
  [references/importers.md](references/importers.md)).
- `console.log(...)` and `console.error(...)` — the same import log with a `JavaScript:` prefix.
- `strftime(format, date)` — date formatting, e.g. `strftime('%Y-%m-%d', new Date())`.
- `_` — the underscore.js utility library (`_.groupBy`, `_.uniq`, `_.sortBy`, …).
- ISO 8601 `Date.parse`, e.g. `Date.parse('2015-05-21T15:24:02.225+0300')` returns milliseconds since epoch.

## Reference documentation

| Topic | File |
|---|---|
| `application.js` manifest and source params | [references/application-manifest.md](references/application-manifest.md) |
| `schema_definition.js` dimensions, cubes, measures | [references/schema-definition.md](references/schema-definition.md) |
| `api.js` requests, pagination, SQL queries, selection | [references/api.md](references/api.md) |
| `importer.js` and `importers/*.js` import logic | [references/importers.md](references/importers.md) |
| `dwh` dimension and measures tables | [references/dwh-tables.md](references/dwh-tables.md) |
| `sample_reports.js` reports and dashboards | [references/sample-reports.md](references/sample-reports.md) |
| `icon.svg` and `icon_dark.svg` | [references/icons.md](references/icons.md) |
| `test/*_test.js` tests and mocked responses | [references/testing.md](references/testing.md) |

## Recommended build order

1. **Explore the source API** — which objects exist, how they are paginated, which timestamps allow
   incremental imports, how authentication works.
2. **`application.js`** — the connection form and authentication first, so that the connection can be
   tested before anything else is written. Prefer a service account type (`google_service_account`,
   `oauth2_client_credentials`, `aws_sigv4`, ...) over `oauth2` when the data belongs to an organization
   rather than to one user: no user has to authorize the connection.
3. **`api.js`** — one function per source endpoint, returning plain data. Add `testConnection`, when the
   user should select what to import, `getSourceSelections`, and when the user should select custom fields,
   `getAvailableCustomFields`.
4. **`schema_definition.js`** — design dimensions first (what the user wants to group by), then the cube
   measures (what the user wants to count and sum), then calculated members.
5. **`importer.js` and `importers/*.js`** — import dimension members, then fact rows per imported object,
   with a `try`/`catch` reporting the failure of one object. Add incremental import max params once the full
   import works.
6. **`test/*_test.js`** — write a test for each function as you add it and run the tests on the server
   with `eazybi custom-source-apps run-tests <dir>`, they are the fastest way to see what an
   implementation change does (see [references/testing.md](references/testing.md)).
7. **`sample_reports.js`** and **icons** last.

## Install and test

**Use the [eazyBI CLI](https://docs.eazybi.com/eazybi/set-up-and-administer/eazybi-cli) version 0.4.0 or later for
every step below.** It packages a local implementation directory, runs the tests, installs into the scope you
name, and drives the imports — one command each, instead of hand-built `curl` calls. Check the installed
version with `eazybi --version`. Log in once with the scopes of the scope you install into
(`eazybi auth login -p dev --host … --scopes read,write,admin:read,admin:write,admin:all_accounts`).

```sh
eazybi custom-source-apps run-tests ./github      # tests only, installs nothing
eazybi custom-source-apps install ./github -a 42  # into one test account
eazybi source-applications start-import 7 -a 42         # repeat the import after a change
eazybi source-applications last-import-log 7 -a 42      # what the import logged
```

- Run the tests of the package before installing it — `eazybi custom-source-apps run-tests <dir>`
  runs `test/*_test.js` against mocked source system responses in a throwaway account and reports the
  failures and the import log, without installing anything. It exits non-zero when a test failed, so it
  is the check to run before every install (see [references/testing.md](references/testing.md) for the
  endpoint behind it and the scopes each scope needs).
- An app is installed with `eazybi custom-source-apps install` from a local directory or ZIP file,
  from a GitHub repository URL (`https://github.com/owner/repository`, narrowed with `--ref` — a tag, a commit
  or a branch, also a branch with slashes — and `--subdir`), from a Bitbucket Cloud repository URL
  (`https://bitbucket.org/workspace/repository`, optionally with `/src/ref/subdirectory/`), or — in a
  development environment — from a directory on the server (`--server-directory`). Instance admins manage
  global apps; site admins and account owners manage the apps of their own site (`--site`) or account
  (`-a`) when that is enabled for them.
- The manifest `key` declares the application type and is never given when installing. Reinstalling an
  existing app from a package with a changed `key` fails.
- A new install is in **draft** status: it is selectable only by the admins or the owner of the scope it
  was installed into, so it can be tested before `eazybi custom-source-apps enable <id>` publishes
  it to everybody.
- Always test imports in a new empty test account: a schema change can require dropping and recreating
  the account cube tables.
- Install into the **account** scope first (`-a <account id>`): an account app overrides the site and
  global app of the same application type, so a change is proven in one account before it is published
  wider.
- Reinstall from the origin (`eazybi custom-source-apps update-from-origin <id>`) to pick up
  implementation changes. The manifest `version` is not incremented for every reinstall — it changes
  when a new version is published (see the `version` option in
  [references/application-manifest.md](references/application-manifest.md)).
- The source application itself is created in the UI (source params, source selection, import options)
  and the first import is started there. Every later import is CLI work:
  `eazybi source-applications start-import <id>` (`--reimport` for a full reimport), poll
  `eazybi source-applications show <id> --json=status,status_message,error_message`, and read
  `eazybi source-applications last-import-log <id>` — the import log carries everything the
  implementation logged with `logger.*` and `console.*`, plus the profiling results of each importer.

## Hard rules to design against

- **No SQL against the eazyBI data warehouse.** Schema expressions use the `expr.*` helpers, all table
  conditions are attribute equality objects. Column and table names must exist in the schema. The database of
  the user is queried with `source.sql` only, and only when the manifest declares its `sql` connection.
- **Secrets never reach JavaScript.** `secret` source params and OAuth tokens are applied to the HTTP
  connection by eazyBI, the database password to the SQL connection; reading them from JavaScript is not
  possible. The other source params are readable
  with `source.getSourceParam(key)` and `source.getSourceParams()`.
- **Requests are limited to the declared hosts.** Only the base URL host and the manifest `additionalHosts`
  can be requested with the credentials, always through `source.*` functions, and a redirect to any other host
  is followed without them (a redirect of a request with a body to another host is an error). A public
  document is requested without the credentials (`sendAuth: false`), which allows the manifest `publicHosts`
  as well. Every host an app reaches is declared in one of the two manifest lists.
  The SQL connection is read-only and every query is rolled back.
- **Everything stored or displayed is sanitized.** Warnings and status messages are HTML escaped, source
  selection labels and names are truncated plain text that eazyBI escapes where it displays them.
- **Every stored timestamp goes through a `dwh` time function.** `dwh.lookupTimeId`, `dwh.datetimeValue` and
  `dwh.dateValue` convert into the Time dimension time zone the user chose on the import options page. A raw
  source timestamp written into a `datetime` column disagrees with the Time dimension of the same value, and
  with what the built-in applications show for the same object. A value the source gives as a date without a
  time (`"2026-01-12"` — a due date, a release date, a birthday) goes through `dwh.lookupDateId`, which looks
  it up as written — converted as midnight UTC it would fall on the previous day in a time zone behind UTC.
- **One `importRows` call per import scope**, at most 10000 rows per call.
- **A failed object is reported, not swallowed.** `source.warnings.addObjectFailure(key, message)` under a
  stable object key keeps the incremental mark back so that the next import retries the object; an error
  caught and not reported leaves its data missing until the next full reimport.
- **No shared JavaScript state between parallel workers** — each worker owns its own engine. Loop over the
  imported objects with `forEach` and reach for `forEachParallel` only when one object needs a long chain of
  dependent requests.

## Advanced settings

Users can tune an installed app with advanced settings of the source application. Document the ones your
implementation depends on. An app may also read settings of its own with `source.getAdvancedSetting(key)`
(see [importers.md](references/importers.md)) — document them in the app README. A changed advanced setting
does not clear the incremental import state, so tell the user to run a full reimport after changing a
setting which changes the imported data. The settings that eazyBI itself reads:

| Setting | Default | Effect |
|---|---|---|
| `api_timeout_in_seconds` | 120 | HTTP request timeout |
| `max_concurrent_requests` | 3 | parallel requests of one paginated or multi document fetch |
| `max_pagination_requests` | 10000 | maximum pages of one `getAllPaginatedDocuments` or `getAllGraphqlPages` call |
| `max_parallel_workers` | 4 | upper limit of `forEachParallel` workers |
| `object_warning_limit` | 100 | reported object failures tolerated before the import fails |
| `import_timeout_limit` | 14400 | maximum duration of one import in seconds (4 hours) |
| `max_wait_seconds` | 3600 | how long an import may wait in `source.waitFor` for an asynchronous export |
| `max_download_size` | 1073741824 | maximum size of one `source.downloadFile` file, and of what a ZIP archive unpacks to (1 GB) |
| `max_json_file_size` | 67108864 | maximum size of a downloaded file read with `format: "json"` (64 MB) |
| `max_document_size` | 67108864 | maximum size of the response body of one fetch function request (64 MB) |
| `ssl.verify` | true | verify the TLS certificate of the source system; `false` accepts a self-signed certificate |
| `javascript.compiled_mode` | true in production, false elsewhere | compile the JavaScript of the app in the importer engines |
| `sql.connect_timeout` | 20 | seconds to wait for the SQL database connection |
| `sql.query_timeout` | 600 | seconds a SQL query may run before it is canceled |
| `sql.read_timeout` | query_timeout + 30 | socket read timeout of the SQL connection in seconds |
| `sql.max_select_rows` | 10000 | rows one `source.sql.selectAll` call may return |

A production import compiles the JavaScript of the app by default, because a large import is about 20%
faster with it; a development import and a test run in every environment interpret it, so that the
JavaScript of an app which is being written is evaluated the same way as in its test runs.
`javascript.compiled_mode` overrides the default either way. Compiling runs a computing function about
twice as fast, but it costs about 25 ms more to build every importer engine of the import — the main
importer engine and one per `forEachParallel` worker; the api engine which tests the connection and builds
the source selection stays interpreted — and it gains nothing for an import which waits for requests and
for the database. Unbounded recursion then fails the import with an error instead of being stopped at a
fixed depth. An app whose import time goes into the source system requests and the dwh table calls may be
faster with the setting turned off in production; read the profiling results of the import log both ways
before changing the default.

## Before you install

- Every file defines the namespace derived from its name; `importer.js` defines `app.importer.importAll`.
- Manifest: valid `key` (lowercase letters, digits, underscores), `name`, every source params field has a
  `key`, a supported `type` and a `label`; `select` fields have `options`; the auth block has all keys of
  its type.
- Schema: at least one cube; several cubes are all named and declare their `dimensions`; no dimension table
  name contains `_measures`; no two tables derive the same physical table name; expressions use `expr.*`.
- Icons contain no scripts, event handlers or external references.
- Sample report and dashboard names are stable — they are matched by name on reimport.