Marketo input plugin for Embulk
embulk-input-marketo is the gem preparing Embulk input plugins for Marketo.
- Lead(lead)
- Activity log(activity)
- Lead by list(all_lead_with_list_id)
- Lead by program(all_lead_with_program_id)
- Campaign(campaign)
- Assets Programs (program)
This plugin uses Marketo REST API.
Overview
Required Embulk version >= 0.8.33 (since 0.6.0).
- Plugin type: input
- Resume supported: no
- Cleanup supported: no
- Guess supported: no
Install
$ embulk gem install embulk-input-marketo
Configuration
API
Below parameters are shown in "Admin" > "Web Services" page in Marketo.
Base configuration parameter
All target have this configuration parameters
| name | required | default value | description |
|---|---|---|---|
| target | true | Marketo targets | |
| account_id | true | Marketo Muchkin id | |
| client_id | true | Marketo REST client id | |
| client_secret | true | Marketo REST client secret | |
| marketo_limit_interval_milis | false | 20 | Marketo have limitation of 100 calls per 20 second. If REST API calls are failed they will wait this amount of time before retry |
| batch_size | false | 300 | Token paging batch size. Some REST API support batch |
| max_return | false | 200 | Max return for Endpoint that use offset paging |
Bulk extract target configuration parameter (Lead and Activity)
All bulk extract target use this configuration parameter
| name | required | default value | description |
|---|---|---|---|
| from_date | true | Import data since this date. Example: 2017-10-11T06:43:24+00:00 | |
| fetch_days | false | 1 | Amount of days to fetch since from_date |
| polling_interval_second | false | 60 | Amount of time to wait between pooling job status in second |
| bulk_job_timeout_second | false | 3600 | Amount of time to wait for bulk job to complete in second |
| incremental | false | true | If incremental is set to true, next run will have from_date set to the previous to_date(calculated by from_date + fetch_days) |
| incremental_column | false | createdAt | Column use to filter from_date and to_date |
Lead
Lead target extract all Marketo leads, it use Marketo bulk extract feature. Configuration include bulk extract configuration.
target: lead
Configuration:
| name | required | default value | description |
|---|---|---|---|
| use_updated_at | false | false | Support filter with updateAt column, but not all Marketo Account have the feature to filter by updatedAt, updatedAt don't support incremental ingestion |
| included_fields | false | null | List of lead fields to included in export request sent to Marketo, can be used to reduce the size of BulkExtract file |
Schema type: Dynamic via describe lead endpoint.
Incremental support: yes
Range ingestion: yes
Activity
Activity target extract all Marketo activity log. Configuration include all bulk extract configuration
target: activity
Schema type: Static schema
Incremental support: yes
Range ingestion: yes
Filter by specific activity type ids: yes. See #95
Campaign
Campaign extract all campaign data from Marketo
target: campaign
Schema type: Static schema
Incremental support: no
Range ingestion: no
Lead by list
Extract all Lead data including lead's list id
target: all_lead_with_list_id
Configuration:
| name | required | default value | description |
|---|---|---|---|
| included_fields | false | null | List of lead fields to included in export request sent to Marketo, can be used to reduce request, response size |
Schema type: Dynamic via describe leads. Schema will have 1 addition column name listId that contain the id of the list the lead belong to
Incremental support: no
Range ingestion: no
Lead by program
Extract all Lead data including lead's program id
target: all_lead_with_program_id
Configuration:
| name | required | default value | description |
|---|---|---|---|
| included_fields | false | null | List of lead fields to included in export request sent to Marketo, can be used to reduce request, response size |
Schema type: Dynamic via describe leads. Schema will have 1 addition column name listId that contain the id of the list the lead belong to
Incremental support: no
Range ingestion: no
Assets programs
Get Assets Programs by Query Tag type, Date range or all if no query by specified.
target: program
Configuration:
| name | required | default value | description |
|---|---|---|---|
| query_by | false | null | Get assets programs by query, supported values date_range, tag_type leave unset to fetch all programs |
| earliest_updated_at | false | null | Required if query by date_range is selected. Exclude programs prior to this date. Must be valid ISO-8601 string |
| latest_updated_at | false | null | Required if query by date_range is selected. Exclude programs after this date. Must be valid ISO-8601 string |
| filter_type | false | null | Optional value send with query by date_range is selected to filter out the result from Marketo. Supported values id, programId, folderId, workspace |
| filter_values | false | null | Set the values associated with filter_type |
| tag_type | false | null | Required if query by tag_type is selected. Type of program tag |
| tag_value | false | null | Required if query by tag_type is selected. Value of the tag |
| report_duration | false | null | Amount of milliseconds to fetch from earliest_updated_at. If incremental = true this value will automatically calculated for the first run by latest_updated_at - earliest_updated_at |
| incremental | false | true | If incremental is set to true, next run will have earliest_updated_at set to the previous latest_updated_at + report_duration. Incremental import only support by query date_range |
Schema type: Static schema
Incremental support: yes (Query by date_range only)
Range ingestion: yes
Example
For lead, you have partial-config.yml like below:
in:
type: marketo
target: lead
account_id: ACCOUNT_ID
client_id: CLIENT_ID
client_secret: CLIENT_SECRET
from_date: 2017-09-01
fetch_days: 1
out:
type: stdout
You can run embulk guess partial-config.yml -o lead-config.yml and got lead-config.yml. lead-config.yml includes a schema for Lead.
Next, you can run embulk preview lead-config.yml for preview and embulk run lead-config.yml for run.
Example of Assets Programs config
in:
account_id: ACCOUNT_ID
client_id: CLIENT_ID
client_secret: CLIENT_SECRET
target: program
type: marketo
query_by: date_range
filter_type: folderId
filter_values:
- 2598
- 1001
earliest_updated_at: 2018-08-20T00:00:00.000Z
latest_updated_at: 2018-08-31T00:00:00.000Z
incremental: true