Automation: workflow integration

The Automation workflow platform supports self-service file processing. Your organization’s account will be configured with one or more workflows defined to meet your data processing needs. In addition to the My TargetSmart web interface, developers can programmatically execute these workflows by posting files to TargetSmart’s SFTP server.

Automation workflows are designed to your organization's unique file processing needs. For our standard list matching solution, see SmartMatch and its API.

Overview

Each Automation workflow is linked to a TargetSmart data transfer account provisioned for your organization. The SFTP protocol is used to access this data transfer account. A top-level directory named automation will exist in your SFTP account. The Automation system polls this directory searching for workflow execution requests. An uploaded trigger file initiates file processing. These trigger files must end with .job.xml and be a valid XML document as described below.

When the Automation system recognizes a .job.xml file, it will attempt to parse the document. If the contents are parsed successfully, the file will be renamed to end with .job.xml.good. If the contents cannot be parsed successfully, the file will be renamed to end with .job.xml.bad.

The naming of the uploaded files is your choice. The only restriction is the file name must end with .job.xml and correctly reference workflow input and output file names also stored in the automation SFTP account directory. All files should be stored in the automation directory and referenced with the relative path basename. The system does not currently support nested files/directories.

The Job XML document references the workflow input file that you wish to process using its relative path basename. This input data file should be fully uploaded before subsequently uploading the corresponding job XML trigger file that references it.

Preparing your input file

You will need to prepare your input file to meet the requirements of the workflow you are executing. The input file format requirements are published in the My TargetSmart Automation Console for each workflow provisioned for your organization.

Job XML format

A job XML file is simply a XML document that includes the parameters needed to start an Automation job.

  <job>
    <inputfile>test.txt</inputfile>
    <outputfile>output.txt</outputfile>
    <jobtype>geocoding</jobtype>
  </job>

An XML declaration, i.e., <?xml version="1.0" encoding="UTF-8" ?>, is optional.

The inputfile element text should be the name of an existing file in the same directory. The outputfile element text should be the name of the output file to be written to the same directory. The jobtype element text should be a valid workflow name as provisioned for your organization. Access the My TargetSmart Automation console to find the valid workflow names for your account. The workflow names are presented as drop-down selection in the first step of the console wizard.

Required elements

The top-level element must be <job>. The <job> element must contain the following required child elements:

  • <inputfile>: The input file to process. This file must be in the same directory as the job XML file and should only be the basename (not full path) of an existing file.
  • <outputfile>: The output file to eventually write. The output file will be written to the same directory where the job XML file is found. The name used here should just be a basename (not full path).
  • <jobtype>: A string representing the type of job to run. Our staff defines the strings you can use here. You can use the web interface to get the list of valid job types or ask our staff.

Optional elements

The <job> element can contain the following optional child elements:

  • <args> : An optional list of job argument elements to parameterize the job

The <args> element

The optional <args> element can contain one or more <arg> child elements. An <arg> element represents a job argument that can be used by the job module to configure the behavior of the job at run time. Each <arg> element should have a name attribute and element content that represents an associated value.

For example, all Automation jobs submitted through the web interface always include a set of <arg> element like the following:

<args>
  <arg name="tsmartweb_email">Bob Smith &lt;bob.smith@example.com&gt;</arg>
  <arg name="tsmartweb_user">example_bob_smith</arg>
</args>

Form parameters

A special <arg> element, <arg name="form_custom_json"> can be used to pass in form data collected from the web interface to parameterize the job behavior. The value of this argument should be a valid JSON dictionary object string. This dictionary can contain arbitrarily nested serialized data. The job module has access to this JSON string. Typically this feature is used when the user customizes the parameters of a job through the web interface. If you need to handle custom parameters in this way, ask our staff about the JSON object format to use for your job requirements.

Overriding the status key

Each job is assigned a unique status key. This status key is used by the automation status application to handle status requests, i.e., the status link in the notification emails. By default this key is auto-generated (uuid). You can override the value by using a <arg> with the key __status_key. If this arg used, a string value should be chosen that will not conflict with other jobs.

For example the status key value of d3e7e7478c760539416e6378077785ed can be used to access the job status in HTML:

https://my.targetsmart.com/automation/status?key=d3e7e7478c760539416e6378077785ed

The status is also available as JSON for programmatic status tracking:

https://my.targetsmart.com/automation/status?key=d3e7e7478c760539416e6378077785ed&type=json

Example:

<args>
    .
    .
    .
   <arg name="__status_key">myorg_6afcbf9ec27247fd853926801bc82f76</arg>
</args>

Extra emails

An <arg> with key __emails can be used to specify a list of email addresses that should be copied for notifications. The value should be a string of comma separated email addresses. You can either use this value or send our staff a list of email addresses that should always be copied for Automation notifications for your account.

Start and Finish XML files

An additional integration feature can be enabled by our staff that will write status XML files to your automation directory. This feature is not enabled by default. Contact Client Services to enable this feature.

If enabled, XML job context files will be written to your automation directory to indicate start and finish job state events.

When the Automation framework has validated your .job.xml file and processing has been initiated, a .start.xml file will be created.

Similarly, when a job finishes a .finish.xml file will be created. This file will indicate if the job was successful or ended with errors:

Example .start.xml:

<?xml version="1.0"?>
<jobcontext>
  <id>6205cbf0-1a93-4166-ba47-e567678bdcf0</id>
  <account_name>abc_company</account_name>
  <service_provider>targetsmart</service_provider>
  <inputfile>test.input.txt</inputfile>
  <outputfile>test.output.txt</outputfile>
  <state>running</state>
  <jobtype>test_job</jobtype>
  <module>None</module>
  <pid>1234</pid>
  <jobfile>test.job.xml</jobfile>
  <email_attachment_count>0</email_attachment_count>
  <moreinfo>
    <info key="status_key"
    value="1d361" />
    <info key="status_url"
    value="https://my.targetsmart.com/automation/status?key=1d361&amp;title=TargetSmart%20Automation%20Status" />
    <info key="timezone" value="Eastern Daylight Time" />
  </moreinfo>
  <jobargs />
  <timestamps>
    <timestamp label="starting" time="Thu August 09 11:35:19 AM" />
  </timestamps>
  <errors />
</jobcontext>

Example successful .finish.xml:

<?xml version="1.0"?>
<jobcontext>
  <id>6205cbf0-1a93-4166-ba47-e567678bdcf0</id>
  <account_name>abc_company</account_name>
  <service_provider>targetsmart</service_provider>
  <inputfile>test.input.txt</inputfile>
  <outputfile>test.output.txt</outputfile>
  <state>success</state>
  <jobtype>test_job</jobtype>
  <module>test_job</module>
  <pid>1234</pid>
  <jobfile>test.job.xml</jobfile>
  <email_attachment_count>2</email_attachment_count>
  <moreinfo>
    <info key="status_key"
    value="1d361" />
    <info key="status_url"
    value="https://my.targetsmart.com/automation/status?key=1d361&amp;title=TargetSmart%20Automation%20Status" />
    <info key="input_csv_delimiter" value=" " />
    <info key="input_rec_cnt" value="5200" />
    <info key="output_cnt" value="5200" />
    <info key="output_data_cnt" value="5199" />
    <info key="timezone" value="Eastern Daylight Time" />
  </moreinfo>
  <jobargs>
    <arg name="tsmartweb_user">abc_test_user</arg>
    <arg name="__emails">bill@example.com</arg>
  </jobargs>
  <timestamps>
    <timestamp label="starting" time="Mon August 06 08:13:15 PM" />
    <timestamp label="transferring input file"
    time="Mon August 06 08:13:17 PM" />
    <timestamp label="processing begin"
    time="Mon August 06 08:13:19 PM" />
    <timestamp label="Checking input file"
    time="Mon August 06 08:13:20 PM" />
    <timestamp label="transferring output file"
    time="Mon August 06 08:28:15 PM" />
    <timestamp label="success" time="Mon August 06 08:28:18 PM" />
  </timestamps>
  <errors />
</jobcontext>

Example error .finish.xml:

<?xml version="1.0"?>
<jobcontext>
  <id>6205cbf0-1a93-4166-ba47-e567678bdcf0</id>
  <account_name>abc_company</account_name>
  <service_provider>targetsmart</service_provider>
  <inputfile>test.input.txt</inputfile>
  <outputfile>test.output.txt</outputfile>
  <state>error</state>
  <jobtype>test_job</jobtype>
  <module>test_job</module>
  <pid>1234</pid>
  <jobfile>test.job.xml</jobfile>
  <email_attachment_count>2</email_attachment_count>
  <moreinfo>
    <info key="status_key"
    value="1d361" />
    <info key="status_url"
    value="https://my.targetsmart.com/automation/status?key=1d361&amp;title=TargetSmart%20Automation%20Status" />
  </moreinfo>
  <jobargs>
    <arg name="tsmartweb_user">abc_test_user</arg>
    <arg name="__emails">bill@example.com</arg>
  </jobargs>
  <timestamps>
    <timestamp label="starting" time="Wed July 25 12:14:44 PM" />
    <timestamp label="transferring input file"
    time="Wed July 25 12:14:56 PM" />
    <timestamp label="processing begin"
    time="Wed July 25 12:15:00 PM" />
    <timestamp label="Checking input file"
    time="Wed July 25 12:15:00 PM" />
    <timestamp label="failure" time="Wed July 25 12:15:00 PM" />
  </timestamps>
  <errors>
    <error>Framework caught error</error>
    <error>Error found. Our staff has been notified.</error>
    <error>Incorrect number of fields for record 1. Expected 10.
    Found 40. Parsed fields: ['first_name', 'middle_name',
    'last_name', 'addr1', 'addr2', 'city', 'state', 'zip',
    'matchbackid', '', '', '', '', '', '', '', '', '', '', '', '',
    '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
    '', '', '']</error>
  </errors>
</jobcontext>