Skip to main content

transform API

Client-side ETL with a TypeScript DSL compiled to WASM for zero-copy execution.

npm install @rowops/transform

DSL Builder​

Build transform expressions with a fluent API.

TransformBuilder​

import { TransformBuilder, col, lit, upper, trim } from "@rowops/transform";

const pipeline = new TransformBuilder()
  .derive("email_normalized", upper(trim(col("email"))))
  .derive("full_name", concat(col("first_name"), lit(" "), col("last_name")))
  .cast("age", "number")
  .filter(gt(col("age"), lit(18)))
  .build();

Expression Helpers​

Column & Literal​

import { col, lit, colIndex } from "@rowops/transform";

col("email")         // Reference column by name
colIndex(0)          // Reference column by index
lit("default")       // Literal string value
lit(100)             // Literal number value

Arithmetic​

import { add, sub, mul, div, mod, neg } from "@rowops/transform";

add(col("price"), col("tax"))     // price + tax
sub(col("total"), col("discount")) // total - discount
mul(col("qty"), col("unit_price")) // qty * unit_price
div(col("total"), lit(100))        // total / 100
mod(col("value"), lit(10))         // value % 10
neg(col("balance"))                // -balance

Comparison​

import { eq, ne, lt, le, gt, ge } from "@rowops/transform";

eq(col("status"), lit("active"))   // status == "active"
ne(col("type"), lit("deleted"))    // type != "deleted"
lt(col("age"), lit(18))            // age < 18
le(col("score"), lit(100))         // score <= 100
gt(col("amount"), lit(0))          // amount > 0
ge(col("qty"), lit(1))             // qty >= 1

Logical​

import { and, or, not } from "@rowops/transform";

and(gt(col("age"), lit(18)), eq(col("status"), lit("active")))
or(eq(col("tier"), lit("pro")), eq(col("tier"), lit("enterprise")))
not(eq(col("deleted"), lit(true)))

String Functions​

import { upper, lower, trim, concat, substr } from "@rowops/transform";

upper(col("name"))                        // UPPERCASE
lower(col("email"))                       // lowercase
trim(col("input"))                        // Remove whitespace
concat(col("first"), lit(" "), col("last")) // Concatenate
substr(col("code"), lit(0), lit(3))       // Substring

Null Handling​

import { isNull, coalesce } from "@rowops/transform";

isNull(col("optional_field"))             // Check if null
coalesce(col("nickname"), col("name"))    // First non-null

Type Casting​

import { castToString, castToNumber, castToBool } from "@rowops/transform";

castToString(col("id"))      // Cast to string
castToNumber(col("amount"))  // Cast to number
castToBool(col("flag"))      // Cast to boolean

Transform Operations​

CastOp​

Change column data type.

{
  kind: "cast",
  field: "age",
  target: "number"  // "string" | "number" | "boolean" | "date"
}

RenameOp​

Rename a column.

{
  kind: "rename",
  from: "old_name",
  to: "new_name"
}

DeriveOp​

Create or overwrite a column with an expression.

{
  kind: "derive",
  name: "full_name",
  expression: concat(col("first"), lit(" "), col("last"))
}

FilterOp​

Keep only rows matching a condition.

{
  kind: "filter",
  condition: gt(col("amount"), lit(0))
}

LookupOp​

Join with a lookup table.

{
  kind: "lookup",
  tableId: 1,
  sourceColumn: "country_code",
  targetColumn: "country_name",
  onMissing: "null"  // "null" | "error" | "keep"
}

ConditionalOp​

If-then-else logic.

{
  kind: "conditional",
  name: "tier_label",
  condition: ge(col("spend"), lit(1000)),
  then: lit("premium"),
  else: lit("standard")
}

Pipeline Configuration​

interface TransformPipelineConfig {
  version?: number;
  operations: TransformOp[];
  lookups?: LookupTableConfig[];
}

interface LookupTableConfig {
  tableId: number;
  name: string;
  keyColumn: string;
  onMissing?: "null" | "error" | "keep";
}

Compiler Functions​

validateDSL​

Validate a transform DSL before compilation.

import { validateDSL } from "@rowops/transform";

const errors = validateDSL(pipeline);
if (errors.length > 0) {
  console.error("Invalid pipeline:", errors);
}

compileDSLToBytes​

Compile DSL to binary plan for WASM execution.

import { compileDSLToBytes } from "@rowops/transform";

const planBytes = compileDSLToBytes(pipeline);

Engine Functions​

runTransformPipeline​

Execute a transform pipeline on Arrow IPC data.

import { runTransformPipeline } from "@rowops/transform";
import { resolveBrowserLicense } from "@rowops/import-core";

const { tierGateInit } = await resolveBrowserLicense({
  projectId: "proj_xxx",
  entitlementToken: "eyJ...",
});

const result = await runTransformPipeline({
  inputIpc: arrowBytes,
  config: pipeline,
  lookupTables: [],
  tierGate: tierGateInit,
});

Usage Example​

import {
  TransformBuilder,
  col, lit, lower, trim, concat, gt,
  runTransformPipeline
} from "@rowops/transform";
import { resolveBrowserLicense } from "@rowops/import-core";

const { tierGateInit } = await resolveBrowserLicense({
  projectId: "proj_xxx",
  entitlementToken: "eyJ...",
});

// Build pipeline
const pipeline = new TransformBuilder()
  // Normalize email
  .derive("email", lower(trim(col("email"))))
  // Create full name
  .derive("full_name", concat(col("first_name"), lit(" "), col("last_name")))
  // Cast amount to number
  .cast("amount", "number")
  // Filter positive amounts only
  .filter(gt(col("amount"), lit(0)))
  .build();

// Execute on Arrow data
const result = await runTransformPipeline({
  inputIpc: arrowBytes,
  config: pipeline,
  tierGate: tierGateInit,
});

Tier Restrictions​

FeatureFreeProScaleEnterprise
Basic operations (cast, rename, derive)YesYesYesYes
Filter operationsYesYesYesYes
Lookup operationsNoYesYesYes
Max operations per pipeline52050Unlimited

See Also​