# Merge & Dedup Datasets: Only New Rows, Filter & AI per Row (`automa-flow/dataset-filter-transform`) Actor

Merge several Apify Datasets, remove duplicates across runs and keep only new rows since the last run. Then filter and clean them, or add an AI column on every row with no API key. Dataset cleaner at $1 per 1,000 rows kept; duplicates and seen rows cost $0.02 per 1,000.

- **URL**: https://apify.com/automa-flow/dataset-filter-transform.md
- **Developed by:** [Vadim Bezrukov](https://apify.com/automa-flow) (community)
- **Categories:** Automation, Developer tools, Integrations
- **Features:** Limited permissions
- **Stats:** 11 total users, 7 monthly users, 95.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.98 / 1,000 row kepts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge & Dedup Datasets: Only New Rows, Filter & AI per Row

Merge the Datasets of several scraper runs into one clean Dataset before it goes to a CRM, a sheet or the next Actor: remove duplicates across them, keep only the rows no earlier run wrote, then filter rows by value or date and rename, trim or cast fields. Pick the Datasets in `datasetIds`, or paste a few rows. The default input merges four sample rows into three, because two of them are the same email in another letter case.

Transforms run first, in order. Filters then decide which of those rows are written, and `dedupeBy` leaves out the later copies. The output rows are your fields, not a wrapper. Counts, failures and the next step are in the Key-Value Store record `RUN_SUMMARY`.

A run that keeps every row costs $1 per 1,000 rows: $0.02 for rows read and $0.98 for rows written. A filter that keeps nothing pays only the $0.02 read price, and so do rows left out as duplicates or as seen by an earlier run. There is also a $0.00005 start charge. Platform usage is included in these prices. Set `maxTotalChargeUsd` on API and agent runs. A small test fits in $0.01.

An `llm` transform puts an AI answer in every row: B2B or B2C, an industry, a field pulled out of a description, a sentiment or a translation. It runs through Apify's own `apify/openrouter`, so you need no API key of your own. See [Add an AI column](#add-an-ai-column).

### Merge runs of a scraper

```json
{
  "datasetIds": ["MONDAY_DATASET_ID", "TUESDAY_DATASET_ID", "WEDNESDAY_DATASET_ID"],
  "dedupeBy": ["url"],
  "dedupeNormalize": true
}
```

The Datasets are read in the order you list them, as one stream of rows. The first row of each `url` is written and later copies are left out, also when they sit in another Dataset. `dedupeNormalize` compares the key in any letter case and without outer or repeated spaces, so ` HTTPS://Example.com/p/1` and `https://example.com/p/1` are one row. The rows that are written are not changed. Every transform and filter of this Actor still applies: they run on each row before duplicates are removed.

`RUN_SUMMARY` holds the result: `rowsRead`, `rowsKept`, `rowsDuplicate`, and `sources` with the rows read from each Dataset. Three overlapping Datasets of 40 rows gave `rowsRead` 120, `rowsKept` 80 and `rowsDuplicate` 40 in our test.

- Up to 100 Datasets a run. A Dataset listed twice is read once. `datasetId` still works and is read first when both are set.
- If one of the Datasets cannot be read, the run stops before it writes a row and names it, for example `Source Dataset 2 of 3 (AAAAAAAAAAAAAAAAA) was not found`. It never merges the rest silently.
- `addSourceDataset` adds `_sourceDataset` with the Dataset id to each row before the transforms, so a filter can read it. A `keep` transform has to list it.
- `outputMode` `duplicates` writes only the rows that the default `unique` leaves out, to review them. `count` writes nothing and puts `rowsUnique`, `rowsDuplicate` and `rowsSeenBefore` in `RUN_SUMMARY` for the read price.
- If the platform restarts the run, it continues at the row it had reached, in the Dataset it had reached. No row is written or charged twice.

### Only new rows after every scheduled run

Give the runs a memory name:

```json
{
  "dedupeBy": ["url"],
  "dedupeNormalize": true,
  "newRowsMemory": "daily-leads"
}
```

Save this as a Task. On the scraper (or its Task), open Integrations, connect the Task, and pass `"datasetIds": ["{{resource.defaultDatasetId}}"]` in the integration input (`"datasetId": "{{resource.defaultDatasetId}}"` works the same). Each run then writes only the rows whose `url` no earlier run with the memory `daily-leads` wrote. Rows seen before are counted in `rowsSeenBefore` and pay only the read price. A run that finds nothing new ends as `NO_RESULTS` with an empty Dataset; that is a normal result, not a failure.

- The keys live in a key-value store on your account named `new-rows-memory-daily-leads`. A name with capital letters or underscores gets a short hash added, because storage names take lowercase letters, digits and hyphens. `RUN_SUMMARY.memoryStore` has the name. Delete that store to forget everything.
- Only rows that were written are remembered. A run that stops at `maxItems` or at its charge limit remembers what it wrote, and the next run writes the rest. The keys are saved at least every 15 seconds while rows are written, before a platform migration, and when you abort the run gracefully. An abort without the graceful option stops the run at once: the keys of rows written since the last save (at most the last 15 seconds) are lost, and the next run writes those rows again. No row is skipped either way.
- A row whose `dedupeBy` fields are all empty has no key, so every run writes it.
- Changing `dedupeBy` or `dedupeNormalize` starts a fresh memory, and `RUN_SUMMARY.inputWarning` says so.
- One memory holds up to 5,000,000 keys. Past that the oldest are forgotten, their rows count as new again, and `inputWarning` says how many.
- Two runs with the same memory at the same time both start from the keys saved before them, and both keep the keys they save. A row that is new to both is written by both.
- With an `llm` transform, a row seen before costs no model call when the model does not write the `dedupeBy` fields.
- `"outputMode": "count"` with a memory tells you how many rows are new without writing them or changing the memory.

Without any stored state, pass the Dataset an earlier run of the same rules wrote:

```json
{
  "datasetIds": ["TODAYS_SCRAPER_DATASET_ID"],
  "filterDatasetIds": ["YESTERDAYS_OUTPUT_DATASET_ID"],
  "dedupeBy": ["url"]
}
```

Rows whose key is in a filter Dataset are left out and counted in `rowsSeenBefore`. Filter Datasets are read as they are stored, with no transforms, so they need the `dedupeBy` fields under the names your rows have after your transforms; a Dataset this Actor wrote with the same rules always fits. Their rows pay the read price and are never written.

### Add an AI column

Classify Google Maps leads as B2B or B2C before your CRM. The prompt reads each row's fields through `{{placeholders}}`; the answer goes into `segment`:

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [{"op": "notEmpty", "field": "website"}],
  "transforms": [
    {
      "op": "llm",
      "prompt": "Is this business B2B or B2C? Name: {{title}}. Category: {{categoryName}}. Answer with one word: B2B or B2C.",
      "outputFields": {"segment": ["B2B", "B2C"]},
      "model": "openai/gpt-4o-mini",
      "maxTokens": 16
    },
    {"op": "keep", "fields": ["title", "categoryName", "website", "phone", "segment", "llmError"]}
  ],
  "maxLlmRows": 1000,
  "maxLlmCostUsd": 1
}
```

The filter reads `website`, which the model does not change, so it runs before the model: leads without a website cost no model call. A filter on `segment` would run after it, for example to keep only B2B rows. Put such filters in the same input to send fewer rows to the model. In a merge run (`datasetIds`, `newRowsMemory` or `filterDatasetIds`) with `dedupeBy` on a raw field such as `placeId`, a row whose key was already written or seen also costs no model call.

After each scraper run: on the Google Maps scraper (or its Task), open Integrations, connect a Task of this input with `datasetId` left empty, and pass `"datasetId": "{{resource.defaultDatasetId}}"` in the integration input.

1,000 leads, all kept with an answer: $0.02 read + $0.98 kept + $1.00 for AI rows = $2.00, plus the model usage that `apify/openrouter` bills to your Apify account (about $0.01 for 1,000 short `gpt-4o-mini` answers on a paid plan; the FREE plan pays 10x). `RUN_SUMMARY.llmUsageCostUsd` shows what the proxy reported for your run.

Pull fields out of job descriptions. With `outputFields`, each answer is checked against its type and written to its own column; an answer that does not fit is a row failure with `llmError`, never a guessed value:

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "llm",
      "prompt": "From this job post, give the salary as written, whether the job is remote, and the seniority.\n\nTitle: {{title}}\n\n{{description}}",
      "outputFields": {
        "salary": "string",
        "remote": "boolean",
        "seniority": ["junior", "mid", "senior", "lead", "unknown"]
      },
      "system": "Answer from the post only. Write \"none\" for a salary the post does not give.",
      "maxTokens": 128
    }
  ]
}
```

Translate review text, and skip the rows that are empty:

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [{"op": "notEmpty", "field": "text"}],
  "transforms": [
    {
      "op": "llm",
      "prompt": "Translate this review into English. Reply with the translation only.\n\n{{text}}",
      "to": "textEnglish",
      "maxTokens": 512
    }
  ]
}
```

How the `llm` step works:

- `to` takes a text answer, trimmed. `outputFields` takes 1-10 typed answers: `"string"`, `"number"`, `"integer"`, `"boolean"` or a list of allowed answers. Each row the step reaches also gets `llmError`: null, or the code of why it has no answer.
- `model` is any OpenRouter model id; the default is `openai/gpt-4o-mini`. Tested on this Actor: `openai/gpt-4o-mini`, `openai/gpt-4.1-nano`, `openai/gpt-4.1-mini`, `google/gemini-2.5-flash-lite` and `google/gemini-2.5-flash`. Another id runs at that model's own price, and `RUN_SUMMARY.inputWarning` says so.
- `maxTokens` is 256 by default and at most 2,048, the longest answer `apify/openrouter` returns on the FREE plan. `temperature` is 0 by default. `system` is sent as written before every prompt.
- `onError`: `keep` (default) writes the row with null answers and the code in `llmError`; `drop` leaves the row out, charged as read only.
- Placeholders read nested fields such as `{{metadata.title}}`. A missing field is empty, an object or list is inserted as JSON, and a field that no row of the first batch has is named in `RUN_SUMMARY.inputWarning`. A filled-in prompt over 20,000 characters is a row failure, `LLM_PROMPT_TOO_LONG`, with no call.
- Limits: `llmConcurrency` (default 8, at most 32) calls at a time, `maxLlmRows` (default 1,000) calls per run, and `maxLlmCostUsd` (default $1) of reported model usage. Rows past a limit get `LLM_LIMIT` or `LLM_COST_LIMIT` and follow `onError`. Calls in flight finish, so usage can pass `maxLlmCostUsd` by a few calls. Rows are sent to the model a little ahead of writing, so when `maxItems`, `maxTotalChargeUsd` or the run timeout stops a run, up to twice `llmConcurrency` answers can go unwritten; they are not charged as AI rows, but their model usage was spent. `llmCache` (default true) answers identical filled-in prompts with one call; those rows are still AI rows.
- One call waits at most 90 seconds and is not sent again after a timeout, because a slow generation can still bill model usage. 429 and 5xx answers are retried up to 3 times. If the first 20 rows sent to the model all fail with the same code, for example an unknown model id, the run stops as `FAILED` before more calls, and `nextAction` says what to change.
- One `llm` transform per run. Filters that read fields the `llm` step and later transforms do not change run before the call. In `OR` mode that happens only when every filter can.

### Try the sample

Leave the four prefilled rows, `dedupeBy` `email`, `dedupeNormalize` and the trim step. Run it. The Dataset should contain three rows:

```json
{"company": "Northwind", "email": "Ada@Example.com", "run": "monday"}
{"company": "Contoso", "email": "kim@contoso.example", "run": "monday"}
{"company": "Fabrikam", "email": "lee@fabrikam.example", "run": "tuesday"}
```

The Tuesday `Northwind` row has the same email in another letter case, so it is left out. It is still counted in `RUN_SUMMARY.rowsRead` and `rowsDuplicate`.

For a real merge, pick the source Datasets. If the sample rows are still filled in, the Datasets win and those rows are ignored. `RUN_SUMMARY.inputWarning` says so. `maxItems` is 100 in the demo. Delete it to process the whole Dataset. If at least 50 rows are read and every one fails the same rule, the run stops before you are charged. The same stop happens when those rows all become empty objects, which is what a `keep` of the wrong field names does. A smaller source with a bad cell finishes as `PARTIAL` with the failure codes. When fewer than 50 rows come out of `keep`, `drop` or `rename` with no fields left, those rows are not written and are not charged as kept; `RUN_SUMMARY.skipCodes.EMPTY_ROW` counts them. If a rule names a field that none of those rows have, the run continues and `RUN_SUMMARY.inputWarning` names the field, so a typo or a leftover sample rule is easy to spot. The same warning names the `cast` to add when a number or boolean filter reads a field that holds `"12.50"` or `"true"` as text.

### Run it again

The useful second run is the next export, not a second copy of the same rows. Save a Task with the rules and leave `datasetId` empty:

```json
{
  "filterMode": "AND",
  "filters": [{"op": "equals", "field": "status", "value": "open"}],
  "transforms": [
    {"op": "trim", "field": "company"},
    {"op": "lowercase", "field": "email"},
    {"op": "rename", "field": "company", "to": "companyName"}
  ]
}
```

On the scraper, open Integrations, connect this Task, and pass `datasetId` from `{{resource.defaultDatasetId}}`. The integration fills that placeholder when the scraper finishes. It also adds a `payload` object to the input, which this Actor accepts and does not read. If the integration input has no `datasetId`, the run fails and says what to set, instead of cleaning sample rows saved in the Task. A saved Task stores the text as written, so the placeholder fails the 17-character dataset id check on every start. A schedule starts from the Task input as saved, so it keeps reading one Dataset. Same rules, a new Dataset each time.

### Recipes

Copy one into the input, or into a Task you call from a webhook. Each is a complete input apart from `datasetId`.

#### Clean crawler output for an LLM or RAG pipeline

Keeps the URL, title and Markdown of each page, drops pages without text and writes each URL once. Website Content Crawler stores the title in `metadata.title`: `rename` lifts it to a top-level `title`, and `drop` removes the rest of `metadata`. Because `keep` comes first and names top-level fields, the heavy `html` and `text` fields are never downloaded.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "keep",
      "fields": [
        "url",
        "markdown",
        "metadata"
      ]
    },
    {
      "op": "rename",
      "field": "metadata.title",
      "to": "title"
    },
    {
      "op": "drop",
      "fields": [
        "metadata"
      ]
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "markdown"
    }
  ],
  "dedupeBy": [
    "url"
  ]
}
```

#### Prepare scraped leads for a CRM

Trims the company name, lowercases the email, keeps only leads with an email and writes each email once.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "title"
    },
    {
      "op": "rename",
      "field": "title",
      "to": "company"
    },
    {
      "op": "lowercase",
      "field": "email"
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "email"
    }
  ],
  "dedupeBy": [
    "email"
  ]
}
```

#### Add traffic history to a lead Dataset

Keeps leads that have a website, trims and lowercases it and writes each website once, so the next Actor looks up every site one time.

```json
{
  "datasetId": "YOUR_LEADS_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "website"
    },
    {
      "op": "lowercase",
      "field": "website"
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "website"
    }
  ],
  "dedupeBy": [
    "website"
  ]
}
```

Then run [Website Traffic Checker](https://apify.com/automa-flow/website-traffic-history-checker) with `datasetId` set to this run's Dataset (in an integration, `{{resource.defaultDatasetId}}`). It reads the `website` column, removes scheme, path and `www.`, and returns monthly organic search traffic and the Chrome popularity tier for each site. That Actor runs and charges separately.

#### Keep only the last 7 days on a schedule

`now-7d` is measured from each run's start, so a saved Task keeps filtering the last week. Dates must be ISO 8601, such as `2026-10-01` or `2026-10-01T08:00:00Z`. If no value of that field in the first rows is such a date, for example `Oct 5, 2026` or a Unix timestamp, `RUN_SUMMARY.inputWarning` says so, so an empty result is not mistaken for no new rows.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "gte",
      "field": "postedAt",
      "value": "now-7d"
    }
  ]
}
```

#### Clean prices before filtering by amount

Turns `"$1,299.00"` into `1299` and keeps prices above 10. A cell with no digits, such as `"N/A"`, becomes null and is filtered out instead of failing the row. For prices written with a decimal comma, such as `12,50`, replace the comma with a dot first.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "regexReplace",
      "field": "price",
      "pattern": "[^0-9.]",
      "replacement": ""
    },
    {
      "op": "cast",
      "field": "price",
      "to": "number"
    }
  ],
  "filters": [
    {
      "op": "gt",
      "field": "price",
      "value": 10
    }
  ]
}
```

#### Prepare product images for a bulk download

Keeps active products that have an image URL, names each file after its SKU and writes each SKU and URL pair once. Then pass this run's Dataset ID to [Parallel Bulk File Downloader](https://apify.com/automa-flow/parallel-bulk-file-downloader) with `"urlField": "imageUrl"`, `"fileNameField": "imageName"` and `"metadataFields": ["sku"]`. Use the file extension that matches your images.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "imageUrl"
    },
    {
      "op": "template",
      "field": "imageName",
      "template": "{{sku}}-main.png"
    }
  ],
  "filters": [
    {
      "op": "true",
      "field": "active"
    },
    {
      "op": "notEmpty",
      "field": "imageUrl"
    }
  ],
  "dedupeBy": [
    "sku",
    "imageUrl"
  ]
}
```

The downloader already shares one file and one charge between repeated URLs in the same batch, and it does not charge rows without a URL. This step pays for itself when it removes files you do not need (a file costs $0.004 to download, a kept row $0.001 here, so from about a fifth of the rows), when you need file names built from other fields, or when duplicates span more than one 10,000-row download batch. If every row already holds one URL you want, skip it.

#### Filter scraper output by a nested value

Many scrapers nest values inside objects, such as `price.value` in product rows or `authorMeta.fans` in TikTok rows. A dotted name reads that value. This keeps products rated 4 stars or more that cost at most 50, and writes each product once.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "lte",
      "field": "price.value",
      "value": 50
    },
    {
      "op": "gte",
      "field": "stars",
      "value": 4
    }
  ],
  "dedupeBy": [
    "asin"
  ]
}
```

The same works for TikTok videos from accounts with more than 10,000 followers, with the author name lifted to its own field:

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "rename",
      "field": "authorMeta.name",
      "to": "author"
    }
  ],
  "filters": [
    {
      "op": "gt",
      "field": "authorMeta.fans",
      "value": 10000
    }
  ],
  "dedupeBy": [
    "webVideoUrl"
  ]
}
```

#### Check the rules on 20 rows first

`maxItems` stops after 20 kept rows, so a run on a large Dataset costs about two cents. Read `RUN_SUMMARY` before running it on everything.

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "equals",
      "field": "status",
      "value": "open",
      "caseSensitive": false
    }
  ],
  "maxItems": 20
}
```

### What you can do

| Transform | What it writes |
| --- | --- |
| `trim`, `lowercase`, `uppercase` | Changes a string in `field`, or in each name in `fields`. With one `field` and `to`, writes the result to that new field and keeps the original. Null, a missing field, and other types stay as they are. |
| `rename` | Moves `field` to `to`. A nested `field` such as `metadata.title` moves to the top level. |
| `drop` / `keep` | Removes fields, or writes only the named ones. |
| `cast` | `to` is `string`, `number`, `integer` or `boolean`. `integer` fails a value with a fraction instead of rounding it. Null and a missing field stay as they are. A blank string becomes null for `number` and `boolean`. A value of the wrong type drops the row. Integers with more than 15 digits are a failed cast. |
| `default` | Sets `field` only when it is missing, null or `""`. `0` and `false` stay. |
| `regexReplace` | Replaces matches in the field's text. A number or boolean is read as written and the result is text. An empty or missing `replacement` removes the matches. `\1` and `\g<name>` insert a group; JavaScript `$1`, `$<name>` and `$&` are read the same way when the pattern has that group. Null and a missing field stay as they are. |
| `regexExtract` | Writes the first capturing group that matched, or the whole match when there are no groups. `group` picks one group by number or name instead, and writes null when that group did not take part in the match. A number is read as written. A miss writes null and keeps the row. |
| `template` | Writes `field` from `{{other}}` placeholders. |

| Filter | Matches when |
| --- | --- |
| `equals` / `notEquals` | Values are equal. `true` is not `1`, and the text `"5"` is not the number `5`; `inputWarning` says so when the field holds the other type. A missing field is not equal to null. Text matches letter case unless `"caseSensitive": false`. |
| `contains` / `notContains` | A string contains the text, or a list contains the value. |
| `startsWith` / `endsWith` | The field is a string with that prefix or suffix. |
| `regex` | The pattern is found in the field's text. A number is matched as written; lists and objects never match. |
| `in` / `notIn` | `value` is a list. |
| `gt` / `gte` / `lt` / `lte` / `between` | Both sides are JSON numbers, or ISO 8601 dates when `value` is a date such as `"2026-10-01"`. `"now"`, `"now-7d"`, `"now-24h"` and `"now-30m"` count from the run start. A date without a time is midnight UTC and a time without a zone is UTC, so `lte "2026-10-31"` stops at the start of that day; `inputNotes` says so. Cast a numeric string first. |
| `empty` / `notEmpty` | Missing, null, `""`, `[]` or `{}`. `0` and `false` are not empty. |
| `true` / `false` | The field is a boolean, not the string `"true"`. Cast a text field to `boolean` first. |

`filterMode` is `AND` or `OR`. There is no nested expression tree. A dotted name such as `price.value` or `metadata.title` reads the value nested inside objects, in filters, `dedupeBy`, transforms and `{{placeholders}}`. A top-level field with that exact name wins, so a column literally named `metadata.title` reads as before. A path does not go into lists. A transform changes a nested value where it is, `keep` keeps it under its path, and `rename` moves it to the top-level name in `to`. A field that does not exist yet, and every `to`, is a top-level name. Filters and `dedupeBy` read the field names after the transforms. A name that only `rename` steps moved is read under its new name and noted in `RUN_SUMMARY.inputNotes`: after `status` is renamed to `state`, a filter on `status` reads `state`. A name that a `drop` or `keep` removed, or a renamed field that another transform then changed, fails the run before a row is read, and the message names the field to use.

`dedupeBy` lists the fields that identify a row, such as `["email"]` or `["name", "city"]`. After the transforms and filters, only the first row for each value is written, across all source Datasets in their order. Later rows are counted in `rowsDuplicate` and charged as read, not kept. Values compare exactly unless `dedupeNormalize` is true; it ignores letter case and outer or repeated spaces in text, and leaves numbers as they are, so the number `10` and the text `"10"` stay two keys. Rows whose key fields are all empty are kept. One run checks up to 1,000,000 distinct keys; past that it stops as `DEDUPE_LIMIT` with the rows written so far.

#### Rule format

Each rule is an object with `op`, `field` and the keys its op takes: `value` for comparisons, `pattern` for `regex`, and `to`, `fields`, `replacement` or `template` for transforms. The input form lists them per op.

Text comparisons (`equals`, `notEquals`, `contains`, `notContains`, `startsWith`, `endsWith`, `in`, `notIn`, `regex`) match letter case exactly. Add `"caseSensitive": false` to match any case.

A spelling with only one possible meaning is read in the documented form and noted in `RUN_SUMMARY.inputNotes`: a number written as text for `gt`, `gte`, `lt`, `lte` or `between`, `{"min": 1, "max": 5}` or `"[1, 5]"` for `between`, `values` or a JSON list written as text for `in` and `notIn`, `value` as the `regex` pattern, plain text in `pattern` for a text comparison (text with regex syntax still fails, use `regex`), `"ignoreCase": true` for `"caseSensitive": false`, `"value": null` on `empty`, `notEmpty`, `true` or `false`, `int`, `float`, `bool` or `text` as a cast target, `from` for `rename`, `field` for `keep`, `to` as the value of `default`, a missing `replacement`, JavaScript `$1` in a `replacement`, and a filter or `dedupeBy` on the name a `rename` moved, also when the rename moved an object that holds it. The [changelog](CHANGELOG.md) lists the other spellings that are read. Descriptive keys such as `description`, `comment`, `label` or `id` are ignored.

Anything that could mean two things, such as an unknown key or a comma-separated string for `in`, fails the run before a row is read. The message names the rule, the key and the fix, and `RUN_SUMMARY.inputErrors` lists every rule problem at once, so one retry can fix them all. Each entry has `rule`, `op`, `reason` (`MISSING_KEY`, `BAD_VALUE`, `UNKNOWN_KEY`, `UNUSED_KEY`, `INVALID_REGEX`, `FIELD_REMOVED`, `NO_SOURCE`, `DATASET_ID_ONLY_IN_PAYLOAD`, `PLACEHOLDER_NOT_FILLED`, `UNSUPPORTED`), `key`, the keys the op `accepts` and `message`; a key that does not apply is null. A run that stops on its rules after reading rows has one entry with `RULE_FAILS_EVERY_ROW`, `EMPTY_EVERY_ROW` or `REGEX_TOO_SLOW`.

### How a run ends

The Actor run succeeds when it could read the source and apply the rules, including when nothing matched. Read `RUN_SUMMARY.status`:

| Status | Meaning |
| --- | --- |
| `SUCCESS` | At least one row was written, and no transform failed. With `outputMode` `count`: at least one row passed the rules and was counted. |
| `NO_RESULTS` | The source was read and nothing was written. With `newRowsMemory` or `filterDatasetIds` this is the normal "nothing new" result, and `nextAction` says so. |
| `PARTIAL` | Some rows failed a transform (in a source under 50 rows, possibly all of them), some rows got no model answer (`llmFailed` counts each code), the charge limit stopped the run, or a kept row was read but not written. `rowsUnwritten` counts kept rows that were not stored. `transformFailureSamples` lists up to 20 source indexes. |
| `SKIPPED_CHARGE_LIMIT` | The budget could not pay for one row. Nothing was read. |
| `FAILED` | The input, dataset access, or a charge could not be trusted. An input problem names the rule and the key to fix. |

`rowsSkipped` counts rows that were not written because they are not JSON objects or had no field left after the transforms, and `skipCodes` counts each reason (`NON_OBJECT`, `EMPTY_ROW`). Skipped rows pay only the read price.

A merge run adds `sources` (the rows read from each source Dataset). `filterDatasetIds` adds `filterSources`, `rowsFromFilterDatasets` and `rowsSeenBefore`; those filter rows are part of `rowsRead` and `processedCharged`. `newRowsMemory` adds `rowsSeenBefore`, `newRowsMemory`, `memoryStore`, `memoryKeysAtStart` and `memoryKeysSaved`. An `outputMode` other than `unique` adds `outputMode` and `rowsUnique`, the first rows of each key that were not written; in `duplicates` mode the written duplicates are `rowsKept`. `dedupeNormalize` is listed when it is on. A run that uses none of these has the same `RUN_SUMMARY` keys as before.

A run with an `llm` transform adds `llmModel`, `llmRows` (rows the step reached), `llmSucceeded` (with an answer, cache hits included), `llmCacheHits`, `llmCalls`, `llmFailed`, `llmLimitReached`, `llmUsageCostUsd` (the `usage.cost` the proxy reported; null when it reported none) and `aiCharged`. The codes in `llmFailed` and `llmError` are `LLM_TIMEOUT`, `LLM_RATE_LIMITED`, `LLM_UPSTREAM_ERROR`, `LLM_TRANSPORT_ERROR`, `LLM_BAD_RESPONSE`, `LLM_MODEL_INVALID`, `LLM_MODEL_UNSUPPORTED` (the model cannot return `outputFields`), `LLM_REQUEST_REJECTED`, `LLM_AUTH_FAILED`, `LLM_PAYMENT_REQUIRED`, `LLM_TRUNCATED` (the answer hit `maxTokens`), `LLM_EMPTY`, `LLM_REFUSED`, `LLM_JSON_INVALID`, `LLM_FIELD_INVALID`, `LLM_PROMPT_TOO_LONG`, `LLM_LIMIT` and `LLM_COST_LIMIT`. `nextAction` names the fix for the most common one. With an `llm` step the Actor writes and charges every 100 rows, and it stops reading new rows about two minutes before the run timeout so the calls in flight can finish.

`stoppedReason` is `COMPLETE`, `MAX_ITEMS`, `BUDGET`, `TIME_LIMIT`, `DEDUPE_LIMIT` or `FAILED`. `TIME_LIMIT` means the run stopped 30 seconds before its timeout with the rows it had written and a `PARTIAL` summary, instead of being cut off; raise the run timeout to process the rest. If the platform restarts the run in the middle of a batch, the Actor checks the charges and rows that batch left behind before it goes on, so rows are not charged twice. When that check cannot tell, the run fails with `BILLING_UNCERTAIN` instead of guessing. Start a new run and check the rows already written. `transformFailureCodes` names the rule, for example `transforms[0]:CAST_FAILED`. `REGEX_TIMEOUT` means that pattern took longer than 50 milliseconds on that row, so the row was dropped and the run continued. `REGEX_FAILED` means the replacement does not fit the pattern.

### Related automa-flow tools

- **Text from a cleaned list of PDF URLs:** [PDF to Text Extractor](https://apify.com/automa-flow/pdf-text-extractor) reads `datasetId` and a URL field. Clean and dedupe the URL Dataset here, then pass this run's Dataset to it.
- **Process a cleaned list of media URLs:** [Bulk FFmpeg Video Converter](https://apify.com/automa-flow/bulk-ffmpeg-media-processor) converts or cuts the media at the URLs of a Dataset. Clean and dedupe that Dataset here first.

Each Actor runs and charges separately. This Actor does not call them.

### Switching from Merge, Dedup & Transform Datasets

An input written for that Actor, either build, runs here as it is: paste the JSON. `RUN_SUMMARY.inputNotes` lists every field that was read under another name. This Actor is not affiliated with that Actor or its author.

| Field in that input | What happens here |
| --- | --- |
| `datasetIds` | The same field. It is a Dataset input here, so the run gets read access to exactly these Datasets. |
| `fields` | Read as `dedupeBy`. No `fields`: the Datasets are merged without removing duplicates. |
| `output` | `unique-items`, `duplicate-items` and `nothing` are read as `outputMode` `unique`, `duplicates` and `count`. |
| `datasetIdsOfFilterItems` | Read as `filterDatasetIds`. |
| `appendDatasetIds` | Every row gets `__datasetId__` with its Dataset id. |
| `fieldsToLoad` | Read as a first `keep` transform of those fields plus the `fields` and `__datasetId__`, so the other fields are not downloaded. |
| `offset`, `limit` | Rows skipped and rows read, counted across the Datasets in their order. |
| `outputDatasetId` | A name is read as `outputDatasetName`. A Dataset id fails: this Actor writes its own run's Dataset or a named one. |
| `nullAsUnique` | Rows whose `fields` are all empty are always kept, which is that Actor's `true`. `false` is read as `true` and noted. |
| `mode`, `parallelLoads`, `parallelPushes`, `uploadBatchSize`, `batchSizeLoad`, `verboseLog`, `customInputData`, `onlyRunsNewerThan`, `onlyRunsOlderThan` | Accepted and ignored. Rows are streamed in batches at the default memory. |
| `preDedupTransformFunction`, `postDedupTransformFunction` | The default function that returns the items unchanged is accepted. Any other JavaScript fails the run before a row is read: this Actor does not run code. Use `transforms` and `filters` for the same change. |
| `outputTo` | `dataset` is accepted. `key-value-store` fails the run. |
| `actorOrTaskId` | Fails the run. Pass the Datasets of those runs in `datasetIds`. |

What is different: the key compares exactly here too, and `dedupeNormalize` is off unless you set it. Rows are charged (see below) where that Actor charges platform usage. Duplicates are found while rows stream, with up to 1,000,000 distinct keys a run.

### What it costs

| Event | Price | Charged when |
| --- | --- | --- |
| `processed-row` | $0.00002 | A row was read and a decision was made. |
| `kept-row` | $0.00098 | A transformed row was written. |
| `ai-row` | $0.001 | A row was written with a usable answer from the `llm` transform. Cache hits count; rows with `llmError` do not. |
| Actor start | $0.00005 | Once per run. |

1,000 rows, all kept: $1 plus the start. 1,000 rows, none kept: $0.02 plus the start. Rows left unread because of `maxItems` or `maxTotalChargeUsd` are not charged. A run without an `llm` transform never charges `ai-row`. A duplicate, a row seen by an earlier run and a row of a filter Dataset are rows read: $0.02 per 1,000. In `outputMode` `duplicates` the written duplicates are kept rows; `count` charges only rows read.

| Job | Rows read | Rows written | Cost |
| --- | ---: | ---: | ---: |
| Merge three Datasets of 40,000 rows with 30,000 duplicates | 120,000 | 90,000 | $2.40 + $88.20 = $90.60 |
| A daily monitor that reads 5,000 rows and finds 150 new | 5,000 | 150 | $0.10 + $0.147 = $0.25 a run |
| Merge 1,250 rows of two runs, drop 250 duplicates, add an AI column | 1,250 | 1,000 | $0.025 + $0.98 + $1.00 for AI rows = $2.01, plus model usage |

Each job also pays the $0.00005 start.

The model itself is not in these prices: `apify/openrouter` bills its usage to the Apify account that runs this Actor, at the model's rate (the FREE plan pays 10x). `maxLlmRows` and `maxLlmCostUsd` bound it; `maxTotalChargeUsd` bounds this Actor's own events.

### Limits

Rows of any size work at the default memory: the Actor reads the source in pages sized from the rows it has seen. For heavy rows, such as crawler output with `html` or `markdown`, put `keep` or `drop` first: those fields are then never downloaded, which makes the run faster and cheaper. This works for top-level names; a leading `keep` or `drop` that names a dotted path downloads whole rows.

This does not run JavaScript or Python that you supply. It does not join or pivot. Only the `llm` transform calls a model, one prompt per row, with no files, images, web browsing or tools. `dedupeBy` removes duplicate rows inside one run, across up to 100 source Datasets; `newRowsMemory` and `filterDatasetIds` carry it across runs. One run checks up to 1,000,000 distinct keys. For a larger merge, split the sources into runs that share one `newRowsMemory` (5,000,000 keys), or pass the first run's output in `filterDatasetIds` of the next. Keeping the last row of a key instead of the first, sorting, joins and file sources (CSV, Excel, sheets) are not part of this Actor. Inline `data` stops at 1,000 rows. A text value longer than 100,000 characters that `trim`, `lowercase`, `uppercase`, a `string` cast, `regexReplace`, `regexExtract`, `template` or the `regex` filter reads drops that row with `FIELD_TOO_LARGE`. A regular expression that runs longer than 50 milliseconds drops that row with `REGEX_TIMEOUT`. After 200 such rows the run stops, because the pattern needs to be simpler. `outputDatasetName` is optional. Leave it empty to use this run's default Dataset, which is empty at the start of every run. A named dataset is appended to, so a schedule writes another copy of the kept rows each time. It cannot be the same Dataset you are reading.

### For an agent

Ask it to merge and dedupe Datasets (`datasetIds`, `dedupeBy`), to return only new rows (`newRowsMemory`), or to filter one Dataset or a small JSON array, with explicit `maxItems` and `maxTotalChargeUsd`. It should build each rule from the op list in the input schema and read `RUN_SUMMARY` before fetching rows. If a run fails on its input, `RUN_SUMMARY.inputErrors` lists every rule to change, with the keys each op accepts. `SUCCESS` means the kept rows are in the Dataset linked from the run output. `NO_RESULTS` means the source was read and the filters excluded everything, or every row was seen before (`rowsSeenBefore`). Do not treat a failed dataset read as an empty export. For an `llm` step, also set `maxLlmRows` and `maxLlmCostUsd`, read `llmFailed` and `llmUsageCostUsd`, and treat a row with `llmError` set as unanswered.

What changed is in the [changelog](https://apify.com/automa-flow/dataset-filter-transform/changelog).

Direct MCP tool: `https://mcp.apify.com?tools=automa-flow/dataset-filter-transform`

Example asks: "Merge datasets `ID_1` and `ID_2`, remove duplicates by url ignoring letter case, and tell me how many were duplicates." "From dataset `DATASET_ID`, keep rows whose status is open, trim company, lowercase email, rename company to companyName, and stop at 100 kept rows."

# Changelog

This Actor's version history is a separate document: https://apify.com/automa-flow/dataset-filter-transform/changelog.md

# Actor input Schema

## `datasetIds` (type: `array`):

Datasets to merge, read in this order as one stream of rows, so Remove duplicates by works across all of them. Up to 100. In an integration use \["{{resource.defaultDatasetId}}"]. A Dataset listed twice is read once. Leave this empty to try the pasted rows below.

## `dedupeBy` (type: `array`):

Field names that identify a row, such as email or url. After the transforms and filters, only the first row for each value is written, across all source Datasets in their order; later rows are counted in rowsDuplicate and charged as read, not kept. Values compare exactly unless Ignore case and spaces is on. Rows whose key fields are all empty are kept. A dotted name such as metadata.title reads a nested value. Up to 10 fields. One run checks up to 1,000,000 distinct values.

## `dedupeNormalize` (type: `boolean`):

Compare the Remove duplicates by values in any letter case and without outer or repeated spaces, so " Ada@X.com" and "ada@x.com" are one row. Recommended for emails, urls and names. Numbers and other values compare as they are. The rows that are written are not changed. Default false.

## `newRowsMemory` (type: `string`):

A name such as daily-leads. Runs with the same name remember the Remove duplicates by values of the rows they wrote, and write only rows that no earlier run wrote. Rows seen before are counted in rowsSeenBefore and charged as read, not kept. Needs Remove duplicates by. 1-63 letters, numbers, hyphens or underscores. Empty means no memory. Changing Remove duplicates by or Ignore case and spaces starts a fresh memory.

## `datasetId` (type: `string`):

One Dataset to read, the same as a single entry in Source Datasets. It is read first when both are set. Saved Tasks and integrations that pass datasetId from {{resource.defaultDatasetId}} keep working. The run needs read access.

## `data` (type: `array`):

Up to 1,000 JSON objects to clean in this run, for a quick test. Ignored when a source Dataset is set. For more rows, write them to a Dataset first.

## `filterDatasetIds` (type: `array`):

Rows whose Remove duplicates by values are in one of these Datasets are not written, for example the Dataset an earlier run of these rules wrote. These Datasets are read as they are stored, with no transforms, are charged as read and are never written. Needs Remove duplicates by. Up to 100.

## `addSourceDataset` (type: `boolean`):

Adds \_sourceDataset with the id of the Dataset a row came from, before the transforms. A keep transform has to list \_sourceDataset to keep it. Default false.

## `outputMode` (type: `string`):

unique (default when empty) writes the first row of each Remove duplicates by value. duplicates writes only the rows that unique leaves out, charged as kept. count writes nothing and reports rowsUnique, rowsDuplicate and rowsSeenBefore in RUN_SUMMARY, charged as read only.

## `filterMode` (type: `string`):

AND keeps a row only when every filter matches. OR keeps a row when any filter matches. No filters keeps every row that transforms cleanly.

## `filters` (type: `array`):

Up to 40 filters, each {"op", "field", ...}. equals, notEquals, contains, notContains, startsWith, endsWith: "value" to compare. gt, gte, lt, lte: "value" as a number, an ISO 8601 date such as "2026-10-01" or "2026-10-01T08:00:00Z", or a relative bound "now", "now-7d", "now-24h" or "now-30m" counted from the run start. A date without a time is midnight UTC. between: "value" \[low, high] using those same numbers or dates. in, notIn: "value" list. regex: "pattern", a Python regular expression. empty, notEmpty, true, false: no value. Add "caseSensitive": false to compare text in any letter case. Comparisons are exact: the number 10 is not the string "10" until a cast transform runs. Missing fields are empty, and they are not equal to null. A dotted name such as price.value or metadata.title reads the value nested inside objects; a top-level field with that exact name wins. Filters read the names after the transforms. A name only renames moved is read under its new name; a name a drop or keep removed, or a renamed field another transform changed, fails the run before a row is read.

## `transforms` (type: `array`):

Up to 40 transforms, applied in order before filters, each {"op", "field", ...}. rename: "field", "to". drop: "field" or "fields". keep: "fields". trim, lowercase, uppercase: "field" or "fields", or one "field" and "to" to write the result to a new field. cast: "field", "to" string, number, integer or boolean (a blank cell becomes null). default: "field", "value". regexReplace: "field", "pattern", "replacement" (empty removes the matches). regexExtract: "field", "pattern", optional "to", optional "group" (a group number or name; by default the first group that matched). template: "field", "template" with {{otherField}}. A transform that cannot be applied drops that row and records a code in RUN_SUMMARY. It does not write a guessed value. A dotted "field" such as metadata.title changes the nested value where it is, and rename moves it to the top-level name in "to". A field that does not exist yet, and every "to", is a top-level name. llm: "prompt" with {{field}} placeholders, and "to" for a text answer or "outputFields" for typed answers; optional "model" (an OpenRouter id, default openai/gpt-4o-mini), "maxTokens" (1-2048, default 256), "temperature" (0-2, default 0), "system" and "onError" (keep or drop). The model runs through apify/openrouter with no API key of your own; its usage is billed to your Apify account. One llm transform per run. Filters that read fields the llm step and later transforms do not change run before the model, so rows they remove cost no model call.

## `llmConcurrency` (type: `integer`):

How many rows are sent to the model at once, 1-32. Default 8. Lower it if the model provider throttles.

## `maxLlmRows` (type: `integer`):

The run sends at most this many rows to the model, 1-1000000. Default 1000. Rows past it get llmError LLM_LIMIT and are kept or dropped by onError. Identical prompts answered from the cache do not count.

## `maxLlmCostUsd` (type: `number`):

Stop sending rows to the model once the usage.cost that apify/openrouter reports adds up to this amount, 0.01-1000. Default 1. Calls already in flight finish, so the total can pass it by a few calls.

## `llmCache` (type: `boolean`):

Default true. Rows whose filled-in prompt is identical share one model call. They still get the answer and are charged as AI rows.

## `maxItems` (type: `integer`):

Stop after this many kept rows. 1 to 10000000. Omit it to process the whole source. Filtered, duplicate, skipped and failed rows do not count toward this cap.

## `outputDatasetName` (type: `string`):

Optional. Empty writes the run's default dataset, which a webhook can pass on as datasetId. A name stores rows in that dataset instead. Leave it empty if the run cannot create datasets.

## `payload` (type: `object`):

Added by an Apify Actor-to-Actor integration. It is accepted so the run can start and is not read. Set datasetId to {{resource.defaultDatasetId}} in the integration input.

## `fields` (type: `array`):

Read as Remove duplicates by. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `output` (type: `string`):

unique-items, duplicate-items or nothing. Read as What to write. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `datasetIdsOfFilterItems` (type: `array`):

Read as Datasets of rows to leave out. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `appendDatasetIds` (type: `boolean`):

true adds **datasetId** with the source Dataset id to each row. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `fieldsToLoad` (type: `array`):

Read as a first keep transform. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `offset` (type: `integer`):

Skip this many source rows, counted across the sources in order. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `limit` (type: `integer`):

Read at most this many source rows. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `outputDatasetId` (type: `string`):

A Dataset name is read as Output dataset name. An id is refused. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `nullAsUnique` (type: `boolean`):

Rows whose key fields are all empty are always kept. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `preDedupTransformFunction` (type: `string`):

Custom JavaScript is not run. Anything but the identity function fails the run. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `postDedupTransformFunction` (type: `string`):

Custom JavaScript is not run. Anything but the identity function fails the run. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `outputTo` (type: `string`):

dataset only. key-value-store fails the run. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `actorOrTaskId` (type: `string`):

Not supported. Pass the run Datasets in Source Datasets. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `onlyRunsNewerThan` (type: `string`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `onlyRunsOlderThan` (type: `string`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `mode` (type: `string`):

Accepted and ignored. Rows are always streamed. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `parallelLoads` (type: `integer`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `parallelPushes` (type: `integer`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `uploadBatchSize` (type: `integer`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `batchSizeLoad` (type: `integer`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `verboseLog` (type: `boolean`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## `customInputData` (type: `object`):

Accepted and ignored. Compatibility with inputs written for another merge Actor. See the README section on switching.

## Actor input object example

```json
{
  "dedupeBy": [
    "email"
  ],
  "dedupeNormalize": true,
  "data": [
    {
      "company": " Northwind ",
      "email": "Ada@Example.com",
      "run": "monday"
    },
    {
      "company": "Northwind",
      "email": "ada@example.com ",
      "run": "tuesday"
    },
    {
      "company": "Contoso",
      "email": "kim@contoso.example",
      "run": "monday"
    },
    {
      "company": "Fabrikam",
      "email": "lee@fabrikam.example",
      "run": "tuesday"
    }
  ],
  "addSourceDataset": false,
  "filterMode": "AND",
  "transforms": [
    {
      "op": "trim",
      "field": "company"
    }
  ],
  "maxItems": 100,
  "outputDatasetName": ""
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

## `billingReceipt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupeBy": [
        "email"
    ],
    "dedupeNormalize": true,
    "data": [
        {
            "company": " Northwind ",
            "email": "Ada@Example.com",
            "run": "monday"
        },
        {
            "company": "Northwind",
            "email": "ada@example.com ",
            "run": "tuesday"
        },
        {
            "company": "Contoso",
            "email": "kim@contoso.example",
            "run": "monday"
        },
        {
            "company": "Fabrikam",
            "email": "lee@fabrikam.example",
            "run": "tuesday"
        }
    ],
    "filterMode": "AND",
    "transforms": [
        {
            "op": "trim",
            "field": "company"
        }
    ],
    "maxItems": 100,
    "outputDatasetName": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("automa-flow/dataset-filter-transform").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "dedupeBy": ["email"],
    "dedupeNormalize": True,
    "data": [
        {
            "company": " Northwind ",
            "email": "Ada@Example.com",
            "run": "monday",
        },
        {
            "company": "Northwind",
            "email": "ada@example.com ",
            "run": "tuesday",
        },
        {
            "company": "Contoso",
            "email": "kim@contoso.example",
            "run": "monday",
        },
        {
            "company": "Fabrikam",
            "email": "lee@fabrikam.example",
            "run": "tuesday",
        },
    ],
    "filterMode": "AND",
    "transforms": [{
            "op": "trim",
            "field": "company",
        }],
    "maxItems": 100,
    "outputDatasetName": "",
}

# Run the Actor and wait for it to finish
run = client.actor("automa-flow/dataset-filter-transform").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupeBy": [
    "email"
  ],
  "dedupeNormalize": true,
  "data": [
    {
      "company": " Northwind ",
      "email": "Ada@Example.com",
      "run": "monday"
    },
    {
      "company": "Northwind",
      "email": "ada@example.com ",
      "run": "tuesday"
    },
    {
      "company": "Contoso",
      "email": "kim@contoso.example",
      "run": "monday"
    },
    {
      "company": "Fabrikam",
      "email": "lee@fabrikam.example",
      "run": "tuesday"
    }
  ],
  "filterMode": "AND",
  "transforms": [
    {
      "op": "trim",
      "field": "company"
    }
  ],
  "maxItems": 100,
  "outputDatasetName": ""
}' |
apify call automa-flow/dataset-filter-transform --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automa-flow/dataset-filter-transform"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ipUO4VRReSlN7N68H/builds/qZOxwQ8qQVnzZ6qmb/openapi.json