Incorporate content from databases, RSS feeds and other web pages

Last Modified:: 20 Mar 2023
User Level:: Power User

Description

With Data Objects, Web Objects and the import-url tag you can use content which is maintained outside of Terminalfour in Terminalfour published pages.

Since the Data Object connects to the data source and retrieves content on each publish, you may see an increase in publish duration. Based on your requirements, you should consider the External Content Syncer or Packages when retrieving large amounts of content from external sources.

However, if your license is limited by content item number, the Content Syncer will import content items into Terminalfour which will increase your content count. The Data Object does not import content items into Terminalfour since content is retrieved and added to the page on publish and so does not add to your content count.

There's more than one way to integrate external data in Terminalfour and the method you choose can depend on the type of data and what you want to do with it.

Database content can be included using the Data Object for databases
RSS content can be included using the Data Object for RSS feeds
Web content from multiple pages (http and https) can be included using the Web object
Web content from one page (http and https) can be included using the Import URL tag

Data Object for Databases

The Data Object connects to an external database each time the site publishes. On publish, the content is pulled from the database and inserted into the HTML of the published page. The presentation of the content can be controlled within the Content Layouts.

For security reasons, the password to the external database is not configured in the Data Object tag, but rather in the Data Object configuration.

Configuration options

A Data Object T4 tag is added to a Content Layout and content is added to the site using the Content Type. The following configuration options are available on the T4 tag (a * indicates that it is required):

Item	Description
method	This should be jdbc.
driver*	Enter the driver that is used to connect to the database. Examples: MySQL: `com.mysql.jdbc.Driver` SQL Server: `com.microsoft.sqlserver.jdbc.SQLServerDriver` If you have not created a Data Source to connect to this type of database before, a database driver may need to be installed.
db-url*	Enter the JDBC URL used to connect to the database. This should be in the format `jdbc:[type]://[HOST][:PORT]:SERVICE` Example for MySQL: `jdbc:mysql://1.2.3.4:8080/smdemo` Example for SQL Server: `jdbc:sqlserver://1.2.3.4:1433;databaseName=smdemo`
username*	Enter the username which has access to the database. This user requires only read access.
table-name	Name of the database table to retrieve content from. This is required unless the sql attribute is used. If content from more than one table is required, this can be a comma-separated a list of table names. e.g. `company,countries`
column-names	Names of the columns to be returned from the table(s) specified in table-name. If the content is from multiple tables, prefix the table name followed by a period before the column name, e.g., `company.id, company.name, price, email_address, department, telephone, countries.name` This is required unless the sql attribute is used.
where	This is an optional conditional statement for the database query and is not required if all data is required from the table selected. Placeholders can be used to replace values with the relevant Content Element. e.g., `where price = $template.price$ and countries.id = company.country`
sql	Rather than specifying the table-name, column-names and where attributes, it is possible to use the sql attribute to contain the full SQL statement. Similar to the where attribute, placeholders can be used to replace values with the relevant Content Element.
max-number	To restrict the number of records displayed by a Data Object, enter the number of records, or the name of the Content Element that controls the number. By default (if not specified), all records are displayed. For further information, refer to an example of the max-number attribute.
formatter*	The name of the Content Layout that will be used to format the data returned from the database.
elem-ident-char	Enter the character that is used to wrap the database column names when creating the Content Layout for the Data Object content. By default a $ symbol is used, however, if the Content Layout for the database content already contains $ symbols (e.g., PHP code), then you can change it here.

Example Data Object Tag

Using a "where" Statement

The "where" option in the Data Object tag is not required and is used, as in a SQL query, to return only data that matches defined criteria.

An example would be:

where="where faculty = 'Science' and countries.id = company.country"

If you would like each Content Item using the Data Object Content Type to display different results, you can apply different criteria to the same Content Element for each Content Item.

For example, if you wanted a Content Item to display all data where the faculty is "Science". Add "Science" to the "Faculty" element in the Content Item then change the "where" option in the Data Object tag to reference the "Faculty" Content Element as $template.Faculty$. This would read as follows:

where="where faculty=$template.Faculty$"

You could also use the same Content Type to publish two Content Items to output the results from both the "Science" and "Humanities" faculties by changing the value in the "Faculty" Content Element.

Using a "sql" attribute

Rather than specifying the table name, columns names and where clauses separately, it is possible to use the sql attribute to enter a full SQL statement. For example:

<t4 type="data-obj" method="jdbc" driver="com.mysql.jdbc.Driver" db-url="jdbc:mysql://localhost/external_db" username="dbusername" sql="select company.id, company.name, price, email_address, department, telephone, countries.name from company,countries where price = $template.price$ and countries.id = company.country" formatter="text/output" />

If you alias columns in your query, you must use the "as" statement; not just white-space.

Creating the Content Layout

To determine the Content Layout for the database query results when the site is published, you must specify the Content Layout that will be used. In your Data Object Content Type, you create a new Content Layout called "text/output" and set the "formatter" option in the Data Object tag to this name.

In the Content Layout, add the HTML to use with the data returned from the database query. You can then call the individually returned columns by putting the column name between two $ signs. An example of this would be:

<h1>$id$</h1>
<p>Name: $example.name$ <br />Email: $email_address$ <br />Dept: $department$<br /> Tel: $telephone$ <br /> Price: $price$ <br /> Country: $countries.name$ </p>

This layout can contain other T4 Tags and HTML code. If the Content Layout for the database content contains $ symbols (e.g. PHP code) then it is possible to specify a different character to be used in order to identify the database fields in the Data Object. An example tag would be:

in which case the "text/output" Content Layout would be:

<h1>+id+</h1>
<p>Name: +example.name+ <br />Email: +email_address+ <br />Dept: +department+<br /> Tel: +telephone+ <br /> Price: +price+ <br /> Country: +countries.name+ </p>

Restrict the number of items displayed

If a database query contains a number of records, and you only want to display the first few records, the Data Object tag can restrict the number of items, max-number="x"

An example tag would be:

The maximum value can also come from a Content Element. An example tag, assuming a Content Element called Number, would be:

Ordering results

Results can be ordered by a column name. For example, to order by a column named "Company Name" you would add order=”Order by company.name”:

<t4 type="data-obj" method="jdbc" driver="com.mysql.jdbc.Driver" db-url="jdbc:mysql://localhost/external_db" username="dbusername" table-name="company,countries" column-names="company.id, company.name, price, email_address, department, telephone, countries.name" where="where price = $template.price$ and countries.id = company.country ORDER BY company.name" formatter="text/output" />

Database Drivers

The driver for the database used by your TERMINALFOUR application (MySQL/MS SQL Server/Oracle) will already be installed. Drivers for other databases will require a separate install in order to sync with them.

To install other drivers, the driver (JAR file) should be accessible to the TERMINALFOUR application i.e., included in the TERMINALFOUR application's classpath. For example, the JAR file could be placed in the Tomcat's lib directory, or another location that is referenced by the Tomcat server.xml configuration file.

For further assistance, please discuss this with your TERMINALFOUR server administrator or contact our support team.

Data Object for RSS

The Data Object connects to an RSS feed each time the site publishes. On publish, the content is taken from the RSS feed and embedded within the HTML of the published page. The presentation of the content can be controlled within the Content Layouts.

Configuration options

A Data Object T4 tag is added to a Content Layout and then content is added to the site using the Content Type. The following configuration options are available on the T4 tag (a * indicates that it is required):

Item	Description
method*	This should be set to a value of `rss`.
rss-url*	The URL of the RSS feed to be accessed. This can be a URL or can reference a Content Type Element where the user enters the URL e.g. $template.url$ . An example of the rss-url options is available.
for-each*	The name of the parent tag for each set of data required. e.g., `item` An example of the for-each options is available.
formatter*	The name of the Content layout to be used to style the published page.
max-number	To restrict the number of records displayed by a Data Object, enter the number of records, or the name of the Content Type element that controls the number. By default (if not specified), all records are displayed. For further information, refer to an example of the max-number attribute.

Example Data Object tag

Configuring the rss-url option

The "rss-url" option should be set to the URL of the RSS feed to be accessed. This can be achieved in one of two ways.

Setting the URL in the Content Layout. This can be used if you would like each Content Item added using the Data Object Content Type to show the same RSS feed. It would look similar to the following:

rss-url="http://www.terminalfour.com/rss/news.xml"

If you would like each Content Item added using the Data Object Content Type to display a different RSS Feed, you can add an element to the Data Object Content Type called "url" and then change the rss-url option in the tag to be similar to the example below. Then when you add content to a Section, you can enter the URL of the feed into the "url" element.

rss-url="$template.url$"

Configuring the "for-each" option

The "for-each" option is related to the parent tag of the information you wish to retrieve. If you look at the following extract from an RSS feed, you can see in this case it is called "item":

<channel>
<title>TERMINALFOUR News</title>
<item>
<title>The Genie is out of the bottle, and at your service - a glimpse into the future</title>
<link>http://www.terminalfour.com/news/</link>
<description>The Genie is out of the bottle, and at your service - a glimpse into the future of student support. In a future not too far away, students are all going to have personal assistants. That's if Deakin University's Genie app is anything to go by.</description>
</item>
<item>
<title>How is Artificial Intelligence revolutionizing HigherEd digital marketing?</title>
<link>http://www.terminalfour.com/news/</link>
<description>Artificial Intelligence (AI) is the technology of the moment. And rightly so, as AI has started to sweep through industries and disrupt them. And it’s not stopping at the front gates of Universities and Colleges.</description>
</item>
</channel>

Creating the Content Layout

To determine the layout of the RSS feed information for the published site, you need to specify a Content Layout to be used. In your Data Object Content Type, create a Content Layout called "text/output" and set the formatter option in the Data Object tag to this name as in the tag example earlier. In this Content Layout, you can create the HTML code you want to wrap around the data in the RSS feed. You can then call the individual tags by putting the tag name between two $ signs. An example would be:

<h2>$title$</h2>
<p>Link: $link$</p>
<p>Description: $description$</p>

Output an attribute of a tag

If you would like to output a tag's attribute value, you write this as $nodeName.attribute[attribute_name]$ in the Content Layout, and it will output the attribute value instead of the node's text.

Example:

So, using the example above, if $entrance.attribute[disabled_access]$ is found in the Content Layout, the value "true" will be output on publish.

Restrict the number of items displayed

If an RSS feed includes 10 items, but you only want to publish the top 3, the Data Object tag can restrict the number of items.

An example tag would be:

The maximum value can also come from a content type element. An example tag, assuming a content type element called Number, would be:

Web Object

A Web Object is a broker similar to the Import URL Broker; however, the Web Object can be configured to follow links on the target URL to the linked web pages too. Each time the site publishes, the content is taken from an http or https location and embedded within the HTML of the published page.

You generate a Web Object in one of two ways:

Using a standard URL
- the same URL is used for all instances of the Web Object
Using a custom URL
- different URLs are used for different instances of the Web Object, depending on input from the user creating content

Configuration Options

A Web Object T4 tag is added to a Content Layout and then content is added to the site using the Content Type. The following configuration options are available on the T4 tag (a * indicates that it is required):

Item	Description
method*	Enter a value of `http`.
start-url*	The target URL for the Web Object. This can contain a variable which looks to an element in the Content Type in order to construct all or part of the target URL. Refer to the example of the standard URL and the example of the custom URL below.
parse-body*	Allows a value of `true` or `false`. If set to true, the Web Object reads only the content between the body tags of the target page. If set to false, the Web Object reads the entire target page. There is also a start-tag and end-tag option below.
start-tag and end-tag	Use specific tags in the code where you want to start displaying content and where you want to end. The quote characters within these values do not need to be escaped. Ensure you use a unique start tag. Including comments in your code can help when choosing a suitable place to start. e.g., `start-tag="<div class="content">" end-tag="</div>"` or `start-tag="<!-- content start -->" end-tag="<!-- content ends -->"`
link-match*	If the Web Object is going to follow links on the target page and import linked pages, use this option to specify the start of the URL which the Web Object can use to match URL strings. The Web Object only imports links starting with the link-match value. If link matching is not required, enter a value of "no-link-match". e.g. `link-match="/careers"`
link-match-import-method*	Currently only allows a value of `subsection`. This instructs the Web Object to create sub-directories to hold the linked content.
max-level-site	Define how far you want to crawl within the content when using the "follow links" option. e.g. `max-level-site="5"`
use-title-attribute	When the Web Object follows links and creates sub-directories for each link, the name of the sub-directory will use the link text. Alternatively, if the url has a title attribute, you can use it by setting the `use-title-attribute="true"`.
cache-expiry*	To decrease the publish time for Web Objects, caching can be enabled. If cache is enabled, the first time the Channel is published, the Web Object creates a folder within the content store to keep a copy of the content. The next time a publish is run on the same Channel, the information is pulled from the content store instead of sending a request to the URL, improving publish performance. After the cache expires, the publish connects to the URL again and restarts the process. The time is set in hours. Setting `cache-expiry="0"` ensures that the cache never expires as it is never refreshed. Setting `cache-expiry="24"` will expire the cache after 24 hours.
object-name	Give the Web Object a name. Each instance of the Web Object it has its own folder; as a result, it is recommended to give it a name. Failing to do so results in the Web Object using the Section ID and Content ID as a name reference.

Use of a standard URL

Create a Content Type; it does not require any additional elements to be added, so the only element is Name. Add a Web Object tag to the Content Layout, and then add content to the site using the Content Type.

In the Content Layout, apply the following code:

After the Content Type has been enabled, you can add content and publish the Channel as normal.

Use of a custom URL

The following example demonstrates a Web Object which is pointed at a person profile management system to output the person details in HTML. The individual person's details can then be published as part of the TERMINALFOUR published page.

The URL for the person profile management system contains a variable for the person's email address:

https://my.othersite.com/here/PEOPLE_SHOW.SHOW_PEOPLE?email=mary.smith@mysite.com will display the details for Mary Smith

https://my.othersite.com/here/PEOPLE_SHOW.SHOW_PEOPLE?email=john.smith@mysite.com will display the details for John Smith

The start-url requires a variable for end users to be able to specify the email address of the person when adding content using the Web object Content type.

Create a Content Type. In this example, the Content Type requires one element to store the email address, called "staffemail".

In the Content Layout, apply the following code:

Add content using the new Content Type. Enter "john.smith@mysite.com" in the element "staffemail" and update the content. Then, add an additional Content Item and enter "mary.smith@mysite.com" into the element "staffemail". Publish the Channel as normal.

The above example replaces the $template.staffemail$ value with the entry in the content's "staffemail" element (mary.smith@mysite.com).

On publish, TERMINALFOUR goes to the url https://my.othersite.com/here/PEOPLE_SHOW.SHOW_PEOPLE?email=mary.smith@mysite.com and makes a copy of the content within the body tags for that page and inserts them into the TERMINALFOUR published page.

The start-url can contain T4 tags. The quote symbols in the T4 tag need to be single quotes ( ' ), as in this example:

start-url="https://my.othersite.com/here/PEOPLE_SHOW.SHOW_PEOPLE?lang=<t4 type='lang-var' default-language='en' en='en' fr='fr'/>"

Import URL

The Import URL T4 Tag allows content to be taken from a web page (http or https) and to be added to the HTML of the published page. To follow links on the specified page and import multiple pages, use a Web Object.

The Import URL tag is added to a Content Layout and then content is added to the site using the Content Type. There are two options for the configuration of the target URL:

Item	Description
url	The URL, including http://. In this case, all instances of the Content Type use the same URL.
element	Name of the Content Type element to use for the URL. In this case, users enter the full URL (including the http://) when adding content using the Content Type. Each instance of the Content Type can use a different URL.

Example Using the URL Attribute

If the following T4 Tag is inserted into a Content Layout, it connects to http://www.terminalfour.com/, reads the page content and includes it in the TERMINALFOUR published page.

Example Using the element Attribute

If the following T4 Tag is inserted into a Content Layout, it connects to the URL specified in the "external link" element, reads the page content and includes it in the TERMINALFOUR published page.