Importing Records

An import is a scheduled workflow with two steps: http_get fetches the records from the other system’s API, and pre_upsert_records creates or updates one Promptless CPT Pages record for each. Every record remembers where it came from, so running the import again updates records instead of duplicating them.

When to use it

• The other system is where staff maintain the data, and the website should show it.
• The system has an API that returns the records as JSON in one response.

For a one-off load, creating the records through the Promptless CPT Pages connector is simpler. FlowMint has no loop over pages of results, so a feed split across many pages needs one fetch-and-import pair per page (see the limits below).

How to build an import

1. Create the record type and its fields in Promptless CPT Pages, so every field key the import writes exists. See Importing and syncing records for the CPT Pages side of an import.
2. Look at the real feed. Create the scheduled workflow, enabled, with just the http_get step (a GET changes nothing), run it once from Tools → Scheduled Actions (see Schedule a workflow), and read the step’s body in the run history. Map from what came back, not from the vendor’s documentation: field names, nesting and the way empty values are written often differ.
3. Add the import step and choose a source, a short key for the system such as recdesk. Never change it later: it is part of every record’s identity, and a new source makes every record new.
4. Set the guards. expect_min_records to a number the real feed never falls below, and max_failure_ratio to the share of bad records you accept.
5. Run it with missing_upstream left at keep. Check the counts and warnings in the step’s output and look at the records on the site.
6. Switch to draft once a few runs look right, if records that disappear from the feed should come off the site.

{
  "trigger": { "type": "schedule", "interval": "daily", "hour": 4, "minute": 30 },
  "steps": [
    { "name": "fetch", "type": "http_get",
      "config": { "url": "https://api.vendor.example/v1/programs?status=active",
                  "headers": { "Accept": "application/json" },
                  "auth": { "credential": "vendor", "scheme": "bearer" },
                  "timeout_seconds": 60 } },
    { "name": "upsert", "type": "pre_upsert_records",
      "config": {
        "post_type": "program",
        "source": "vendor",
        "records": "{{ steps.fetch.body.programs }}",
        "map": {
          "external_id": "{{ item.id }}",
          "title": "{{ item.name }}",
          "excerpt": "{{ item.shortDescription }}",
          "fields": { "event_start": "{{ item.startDate }}", "registration_url": "{{ item.registrationUrl }}" },
          "taxonomies": { "program_category": "{{ item.category }}" }
        },
        "expect_min_records": 10,
        "max_failure_ratio": 0.1,
        "missing_upstream": "keep" } },
    { "name": "report", "type": "log_info",
      "config": { "message": "{{ steps.upsert.created_count }} new, {{ steps.upsert.updated_count }} changed, {{ steps.upsert.drafted_count }} withdrawn, {{ steps.upsert.failed_count }} failed." } }
  ]
}

What happens on each run

• A record that is new in the feed is created and published.
• A record that changed is updated, but only in the keys the map names. An editor’s change to a field the map does not name survives.
• A record that did not change is left alone.
• A record that is no longer in the feed is kept, or with missing_upstream: "draft" switched to draft. Records are never deleted.

The guards make a broken feed fail loudly instead of quietly damaging the site. If fewer than expect_min_records arrive (an empty response, a moved endpoint, an expired token that still answers), the step fails before writing anything. If too many records fail to map (usually a renamed field), the step fails after writing the good ones. Either way nothing is drafted and the run fails, and you get a failure notification at once. Don’t give the import step on_error: retry — a changed feed will not fix itself in 15 minutes. The fetch step can have it, to ride out a short outage at the vendor; see Retries.

Limits and common problems

• A drafted record that returns to the feed stays a draft, unless the map sets status. Publish it by hand, or map "status": "publish", which also republishes records an editor unpublished.
• Blank fields on every record. A map path does not match the feed. The step’s warnings list each missing path and on how many records it was missing.
• Several pages of results. With one fetch and import per page, leave missing_upstream at keep: each import step only sees its own page, so draft would unpublish the records on the other pages.
• Store the API key as a credential (http_vendor in the example) so it is not written into the workflow or its run history; see HTTP steps. Use a read-only key.
• Images. featured_image_url downloads each image once and reuses it; an image that cannot be fetched is a warning, and the record is still written.

Available since FlowMint Workflows 0.8.0.

Related pages

• Record Import Step — Settings, mapping keys, guards and output of pre_upsert_records, the step that creates or updates Promptless CPT Pages records from a list another step fetched.
• HTTP Steps — Call another system's API with http_get, http_post or http_request — settings, how bodies are encoded, what the output holds, and how status codes become errors.
• Schedule a Workflow — Run a workflow hourly, twice daily, daily or weekly in site time, what a scheduled run can read, how to check it is scheduled, and how to run it early.
• Failure Notifications — The alert FlowMint sends when a run fails for good — to Slack if a webhook is stored, otherwise by email — what it contains, and how to route or silence it.
• Importing and Syncing Records — Create or update records from another system by source and external ID, so re-sending never duplicates, unchanged records are skipped and feed images download once.