Schemas

Schemas define the structure and validation rules for imported data. A schema acts as a contract between the data source and the pipeline, specifying expected fields, types, and constraints.

---

Schema Structure

RowOpsSchema

{
  id: string,
  name: string,
  fields: RowOpsSchemaField[],
  maskConfig?: MaskConfig,
  transformConfig?: TransformPipelineConfig
}

RowOpsSchemaField

{
  key: string,           // Internal field identifier
  label: string,         // Display name
  type: FieldType,       // Expected data type
  required?: boolean,    // Whether field must have value
  regex?: string,        // Validation pattern
  enumValues?: string[], // Allowed values
  transforms?: FieldTransform[]  // Field-level transforms
}

---

Field Definitions

Supported Field Types

Type	Description	Validation Behavior
string	Text value	Any string accepted
number	Numeric value	Must parse as number
boolean	True/false	Must parse as boolean
date	Date value	Must parse as valid date
email	Email address	Must match email pattern
phone	Phone number	Format validation
url	URL	Must parse as valid URL

Required Fields

Fields marked required: true must have a non-empty value. Empty strings, null, and undefined are treated as missing.

Regex Validation

The regex property defines a pattern the field value must match:

{
  key: 'zipCode',
  label: 'ZIP Code',
  type: 'string',
  regex: '^\\d{5}(-\\d{4})?$'
}

Enum Values

The enumValues property restricts values to an allowed list:

{
  key: 'status',
  label: 'Status',
  type: 'string',
  enumValues: ['active', 'inactive', 'pending']
}

---

Validation Rules

Schemas define validation declaratively. Rules are applied during the Validate stage:

Rule Type	Defined By	Error Code
Required	required: true	REQUIRED
Type match	type: 'number'	TYPE_MISMATCH
Pattern match	regex: '...'	REGEX_MISMATCH
Allowed values	enumValues: [...]	ENUM_MISMATCH

Rule Evaluation Order

1. Required check (if field is required)
2. Type check (if value is present)
3. Regex check (if pattern defined)
4. Enum check (if allowed values defined)

Validation stops at first failure for each field.

---

Versioning Behavior

Schema Versions

Schemas support versioning:

• Each schema has a version history
• Versions are immutable once created
• Active version is used for new imports

Version Contents

Each version stores:

Property	Description
fields	Field definitions at that version
maskConfig	Masking configuration
transformConfig	Transform configuration

Non-Retroactive Effects

Schema changes do not retroactively affect completed imports:

• Existing import results remain unchanged
• Only new imports use the updated schema
• Historical data is not re-validated

---

Schema Storage

Server-Side

Schemas are stored server-side and versioned.

Client Access

Schemas are fetched during pipeline initialization:

1. Client requests schema by ID
2. Server returns active version
3. Schema is applied locally during pipeline execution

---

Embedded Configuration

Schemas can embed mask and transform configuration:

MaskConfig

{
  maskConfig: {
    fields: {
      ssn: 'redact',
      email: 'hash'
    }
  }
}

TransformConfig

{
  transformConfig: {
    steps: [
      { type: 'rename', from: 'old_name', to: 'new_name' },
      { type: 'coerce', field: 'amount', toType: 'number' }
    ]
  }
}

---

Constraints

Hard Limits

Limit	Value
Max schema fields	500
Max transform steps	20
Max nesting depth	20

Tier Considerations

Some validation and transform features may be tier-gated. Check tier documentation for feature availability.

---

What Schemas Do Not Provide

• Cross-row validation: Each row is validated independently
• Automatic type coercion during validation: Type mismatches are errors, not auto-corrected
• Schema evolution migration: Old data is not automatically migrated to new schema
• Conflict resolution: Schema changes require explicit versioning