Submitting Data in Conduit

There are three approaches to submitting your information to Conduit:

1. Direct edits. You can directly add new records to Conduit through the Edit tab. You can modify or create new records for all of the record types here. 
2. Upload a file. Conduit supports the automatic consumption of files uploaded to the SFTP endpoint and the direct upload and processing of files in the Operations tab. Conduit can handle CSV and XML files based on the mappings provided.
3. Web services. RESTful endpoints allow web services to update Conduit with information and changes - no uploading required, just need an agent or an information source that sends web service messages. These endpoints also support GET methods to retrieve information. For more information please see Conduit Web services and examples. 

Data format

Every record, a line in a CSV or datum object in an XML package that is an object to pass through Conduit, needs to have an action, a record key value, and the data that is needed to create or update the record. Deletions only need the action and record key. 

Example: A deletion CSV might look as follows:

action, username

delete, user1

delete, user2

Example: An XML package to create an enrollment could be:

<?xml version="1.0" encoding="UTF-8"?>

<data>

    <datum action="create">

        <mapping name="course">course_id</mapping>

        <mapping name="user">user_id</mapping>

        <mapping name="role">rolename</mapping>

    </datum>

</data>

If Conduit gets more fields than it has mapped for that kind of record, it will ignore the unmapped information and log a warning that unmapped data was passed. If something is wrong with the record information, Conduit will log an error message detailing the issue. For more information on errors and warnings, please see those documentation pages.

Tip:

You can get a head start on the CSV or XML structure by going to the Conduit Settings tab Mappings sub-tab and then clicking on the Download example mappings buttons to download CSV headers or the XML schema for your configured data mappings.

Actions

Every record for Conduit needs an action, whether in an XML package or in CSV format. For our discussion here we will use the more common CSV format. For details on XML packages via the web services, please visit the Conduit Web services and examples documentation. <link to Conduit Web services and examples>

An action tells Conduit what it should do with the data record that follows. Valid actions depend on the record type but always include create and delete. Information is case sensitive. Conduit actions are all lower case commands. 

• Course Actions: create, update, and delete
• User Actions: create, update, and delete
• Enrollment Actions: create, update, and delete
• Role Assignment Actions: create, update, and delete
• Groups Actions: create, update, and delete
• Group Members Actions:create and delete
• Cohort Members Actions: create and delete

Note:

Conduit cannot delete users that have their authentication type (auth field mapping) set to a different authentication than “conduit”. That authentication tool controls that record and Conduit will not delete them from the site. Conduit does have a setting to automatically suspend a user account that is deleted from Conduit. We recommend using this feature to preserve the user information for reporting. 

Required fields

Required fields are indicated in the Mappings page in Conduit under the Settings tab. For reference here are the required fields for each record type.

User: 

The key record value as defined in the General Settings in Conduit. Either username, email address, or idnumber. In practice, user records will always need a username, an email value, a firstname, a lastname, and a city/town value. 

Course:

The key record value as defined in the General Settings in Conduit. Either shortname or idnumber. In practice, a course will require a shortname and a fullname.

Enrollment:

Enrollments require a coursekey and a userkey. In practice, enrollments also require a role. The role will be identified by the key record value as defined in the General Settings in Conduit. Either shortname or name. 

Role assignments:

Roles do not need to be separated from enrollments typically. This functionality is for assignment of roles into unusual spaces like categories or specific content within a course. It is not advisable to use this for normal enrollments. Contact support or work with your configuration team to determine if this functionality is right for you. 

Roles require a context, moodlekey, userkey, and rolekey to work. The context sets the context level in the system (category, course, content item) and the moodlekey is the internal record value for that context. 

Groups:

Creating groups in a course requires a coursekey and a groupkey, which can then be used to identify the group when handling group enrollment actions. 

Group members: 

Group members can be added once the groups have been created. This requires a coursekey, a userkey, and a groupkey. 

Cohort members:

Adding users into cohorts can be handled via Conduit. This requires a userkey and a cohortkey, which is the id number of the target cohort. 

All the rest of the record data can be added and is set in the Mappings page for each record type. Not all fields need to be mapped and not every mapped field needs to be populated in the message to Conduit. 

Setting defaults

Conduit also supports the setting of default values for record fields for Users and Courses. In the Mappings page, for each mapped field you can set a default value for that record so that when a new record is created but that field is not supplied, Conduit will use the default value provided in the mapping. 

Note:

Custom profile fields (for Users and Courses) are available in the mappings but do not support the default value behavior. When you create a custom field, you can specify the default value there. 

Naming files

Each record type has their own file name and Conduit will look for these specifically named files in SFTP and the upload import tool. Each record type has their own file so there is a naming available for course, user, enrollment, group and group membership, role assignments (separate from enrollments), and cohort membership in the Conduit Settings tab > File sub-tab.

For Example: The Users file name is set to auth.csv for the CSV file of userdata (or auth.xml for files in XML format.) When Conduit finds a file with the name auth.csv it will process that file for user data.

Flexible file name support

Conduit can handle variable file names for each type, Where course.csv files might work as a fixed file name for some processes, others might need support for names that index or change. You can set up Conduit to use simple patterns to match changing file names. In the file name mapping, an asterisk (*) matches any number of characters (including none) and a question mark (?) matches any single character. The (?) is available to allow you to specify the number of characters required in the variable space. 

Example: We uploaded the following files to the SFTP directory; course_.csv, course_1.csv, course_abcdefghij.csv, and course_1667332994.csv.

• Assigning the name for a file as course_*.csv would match all of those files.
• Assigning the name for a file as course_?.csv would match course_1.csv but not the others. 
• Assigning the name for a file as course_??????????.csv would match course_abcdefghij.csv and course_1667332994.csv only as it requires 10 characters after the course_ and before the .csv.

SFTP files

Once the files have been prepared, they can be uploaded to the server directory specified in the Conduit Settings tab > File sub-tab using your favorite SFTP program or the command line SFTP in your operating system. Here are a few SFTP programs you might like to use if you do not have a favorite already.

• FileZilla http://filezilla-project.org/
• WinSCP http://winscp.net/eng/index.php

If you are not sure of your SFTP credentials you can request these in a support ticket.

Conduit cron

Conduit uses two crons to do the tasks it has. There are tasks in the core cron task list for keeping data matched between the Moodle™ records and the Conduit records. There is also a Conduit specific cron that runs every 15 to 20 minutes.When the Conduit cron runs, it will pick up the files and process them. Once complete (success or failure) it will remove the files from the SFTP endpoint and store any problematic files in the quarantine directory specified in the Files settings page unless that behavior has been disabled by the site admin. 

Note:

A Cron is a scheduled job that the servers run to take care of tasks. This job handles a few or many tasks depending on the design and what needs to be done right now. The Conduit cron has a primary job of looking for files to process at the SFTP drop off point.

Processing files

The files are first grouped by type, course, user, etc, then sorted by file name and queued in this prioritized order:

1. course
2. user
3. enrollments
4. role_assign
5. groups
6. group_members
7. cohort_users

Processing is deferred to a queue of Conduit tasks, having one task per file. Each file task has a limited amount of time to run which depends on the site's task processing settings. Each site has also a Conduit task queue capacity which is default to 20. If uploading files directly into the Operations tab, Conduit will still queue these files and apply the priority ordering to multiple files uploaded one after the other. However, it is still best to upload the batch of files in the priority ordering (courses and users before enrollments). 

It is recommended that the "Process any Conduit files" scheduled task is set to run as its default setup so new files are queued as soon as possible.

Examples:

• 3 course files, course_1.csv, course_2.csv, and course_3.csv, are picked up by Conduit. When the cron runs, all files are queued, but course_1.csv is a large file that takes a long time to process, so the next files are skipped from processing. After it's processed, course_2.csv and course_3.csv files are processed by the next Conduit cron run. Any other file types, user, groups, enrollments, etc, will process in the same manner.
• 3 course files, course_4.csv, course_5.csv, and course_6.csv, are all relatively small files and picked up by Conduit. Because all three are small files, they are queued and processed in the same cron run.

Tip: 

When uploading new files via SFTP, never overwrite an existing file; it might contain data that you need to preserve. If using a script, check for an existing file or use the wildcards in the naming that allow you to make a new file name for each upload.

Data field values

Each record value accepts specific kinds of formats and uses validation based on the type of data the field holds. For example, if users are being created with an internal authentication mode (manual or conduit) and the site is using password strength policies, those policies will be applied to the passwords provided to Conduit as well. Creating records with passwords that fail will result in an error message.

For more information on Conduit field values and validation please see the Conduit Mappings article.