Integration with Silktide

How it works

A PHP script placed on the Terminalfour server, is triggered periodically using a cron job. It identifies the list of published files that were updated in the last publish and sends them to the Silktide API. The configuration of this script is done via Terminalfour where the web developer can decide to turn on and off the functionality or determine the defer time for the re-scan.

Features

• The script will send the information in batches of 1000 URLs by default
• The script will run once per publish time; a check is in place to ensure that it does not run twice
• The script can configure the following information:
  • the number of seconds to wait before asking Silktide to rescan the page (allowing a transfer of the files to run to the webserver, if required)
  • the number of files to send in each batch
  • the delay in seconds between batches
• If the script fails, it will pick up all the missing URLs and send them to Silktide the next time it runs successfully
• If the local report does not save for any reason, and the script could not get the latest time when it was run, it will check all URLs within the last 2 hours
• The script has the ability to be disabled at any time by the web developer

Requirements

SilkTide API

• Release: 14th September 2020 (https://monitor.support.silktide.com/guides/change-log/)
• Documentation: https://monitor.support.silktide.com/guides/cms-integrations/silktide-api-for-cms-integration/ (See Publishing paragraph)

Terminalfour

• Version: 8.3+

Server

• PHP 7.2+
• PHP extension: cURL
• Requires time-based job scheduler (Cron for Unix) - it can be set up on one of the web servers but it may be preferable on the Terminalfour server.

Installation Guide

The following installation guide will explain how to set up the Integration. It is divided into steps to step through the process.

Before you start

Before we can start on the installation please ensure that:

• You have a minimum level as Administrator in Terminalfour, otherwise you will not be allowed to perform all action required.
• You have a Silktide API Token as explained in Silktide Documentation: https://monitor.support.silktide.com/guides/cms-integrations/silktide-api-for-cms-integration/ (see Authentication)
• You have downloaded the installation files that will be used during this documentation
• The Section Metadata Content Type is created and in use.
• You should be able to perform SQL queries on the Terminalfour database.
  

Installing the required files

Firstly, the required files will be added:

1. Unzip the installation files downloaded. The zip contains 2 folders, content-types and media-library. We are going to focus on the media-library one for now.
2. The media-library files contains 4 files. If you do not already have a Media Type that handles PHP, JSON and PHAR, it will need to be created, or modify an existing one to include those extentions.
3. The Media Type will require a specific Media Layout that will only output the path to the file (e.g. path/*). If this does not exist, create it as follows:
   
   1. Go to Assets > Content Types and select Media Content Type
   2. Go to the Content Layouts tab and ensure that path/* exists or otherwise create it as follow:
      1. Name: path/*
      2. File Extensions: Default
      3. Syntax type: HTML/XML
      4. Content layout processor: T4 Standard Content
      5. Content Layout Code:
         

         <t4 type="content" name="Media" output="file" modifiers="" />
         

      6. Save it
4. Next, we will add the files to the Media Library.
5. Add a new media category or select a pre-existing one as you wish
6. Add the 4 files as media files:
   1. utils.js: We will see this later on it will be used as part of the programmable layout that publish out the correct config information
   2. silktide-integration.phar: is PHP Archive that contains all the logic of the code
   3. run.php the script responsible for sending the information.
   4. config.json the file that helps to configure the information
7. Once the files are in the Media Library, make a note of the Media ID of each file as they are used later, when configuring the code
8. After that we need to ensure that they get published otherwise the script will not work.  Create a Code content item in a hidden page of the site that contains the Media T4 Tag for each file.
9. Publish the channel.

Once this has been completed, the files should all be published within the Channel's Output Directory.

To ensure that everything so far is working as intended, manually run a publish. If all steps are working correctly you should be able to navigate to the hidden page you created, that will contain the links/paths to the required files. They will be something similar to:

/media-library/categorised/[your-folder-path]/

Please ensure to note done the exact folder (where in the code block above is specified as [your-folder-path].

• You should see the above path
• You should have noted down the 4 Media IDs of the 4 files uploaded
• You should have noted down the Channel Output Directory
• You should have noted down the [your-folder-path] that appeared during the publish report

After publishing out and everything works as expected we can continue to set up the rest of the website. Please proceed as follow:

1. The first thing we are going to set up the config.json properly, to do so Go to Section Metadata Content to create a new element called "Disable Silktide Integration". 
2. First we need to run a SQL query to make sure the Section Metadata Content type is not a system content type. Please see Edit a Section Meta Data Content type.
3. Once the Section Meta Data is a normal content type, you need to ensure you have a list called Yes or if you don't have it create a new one by going to Assets > Lists
   1. Name: Yes
   2. List Items: Yes
      1. Name: Yes
      2. Value: yes
      3. Selected: unchecked
      4. Sublist: None
4. Once you have the list we can used in the new element we need to add inside the Section Metadata Content Type, Go to Assets > Content Types and select the Section Metadata Content type (usually called Section Meta Description):
   1. Name: "Disable Silktide Integration" 
   2. Description: "Check if you want to disable the Silktide Integration (the check should be valid only on the Home)"
   3. Type: Checkbox, List: Yes
   4. Use as filename: false
   5. Required: false
5. Once saved the element you can reset it back as System Content Type following the same documentation above
6. After create a new element we need to add the 2 new content layouts to it, Go back to Assets > Content Types and select again the Section Metadata Content type, this time going to Content layouts
7. Add the following content layout that is responsible to provide the value of the element just added
   1. Name: text/silktide-flag
   2. File Extensions: Default
   3. Syntax type: HTML/XML
   4. Content layout processor: T4 Standard Content

   5. Content Layout Code: 

      <t4 type="content" name="Disable Silktide Integration" output="normal" display_field="value" />

   6. Save it. 
8. Add another content layout, that will be used to set up the config.json information:
   1. Name: text/silktide-config-json
   2. File Extensions: Default
   3. Syntax type: Javascript
   4. Content layout processor: Javascript Content

   5. Content Layout Code: Please copy the content of the file placed inside the installation file zip under /content-types/section-meta-description/text-silktode-config-json.js

   6. Before saving in the code just copied replace at line 19 

      <!-- INSERT SILKTIDE TOKEN -->

       with the SilkTide API Token (see Before you start paragraph) and 

      <!-- INSERT ID-->

        with the utils.js Media ID just created at point 12.a in the previous paragraph.
   7. Save it.
9. Once we have the content layout for the config.js we need a navigation object that retrieve its content and we can place it in the config.json media file added previously. Therefore go to Assets > Navigation.
10. And create 2 new navigation objects, for both content layouts just created.
    1. Navigation Type: Related Content
    2. Name: Silktide Integration: get SilkTide Config JSON for Current Channel
    3. Description: Returns text/silktide-config-json from Section Meta Description
    4. Primary Group: Global
    5. Options: [x] Show pending content in preview and [x] Cache output
    6. Output title: leave blank
    7. Fetch Method: Use Section
    8. Section: Please select the Home section of your channel
    9. Content layout: Use alternate content layout
    10. Alternate content layout: text/silktide-config-json
    11. Before HTML: leave blank
    12. After HTML: leave blank
    13. Save changes
    14. Build the t4 tag for this navigation object and save it in a notepad file for later
11. Now we need the second navigation object for the Disable Silktide Integration with the following attributes:
    1. Navigation Type: Related Content
    2. Name: Silktide Integration: get Flag
    3. Description: Returns text/silktide-flag from Section Meta Description
    4. Primary Group: Global
    5. Options: [x] Show pending content in preview and [x] Cache output
    6. Output title: leave blank
    7. Fetch Method: Use Section
    8. Section: Please select the Home section of your channel
    9. Content layout: Use alternate content layout
    10. Alternate content layout: text/silktide-flag 
    11. Before HTML: leave blank
    12. After HTML: leave blank
    13. Save changes
    14. Build the t4 tag for this navigation object and save it in a notepad file for later
12. Once we have the 2 navigation object we can finish up to set up the Media Files with all references
13. Go to Content > Media Library and select config.json
14. Edit config.json, editor tab replace all content with the new t4 tag created at point 10, and save it
15. Edit run.php, go to the editor tab and for $config['config'] replace 

    <!-- INSERT ID -->

    with the config.json Media ID and $config['library'] replace 

    <!-- INSERT ID -->

    with silktide-integration.phar Media ID, and $config['disable'] with the navigation t4 tag created at point 11
16. Save it all
17. Publish out the results

If everything works as expected you should see a functional PHP script that returns a JSON format and display the Report of the transaction made at a URL like https://www.[yourdomain]/media/[your-folder-path]/run.php.

After this the only outstanding issue is to ensure to add a schedule task on the Terminalfour Application server or one of the web servers that points to server location of the above script [Channel Output Directory]/media/[your-folder-path]/run.php 

After that is done the Integration should be in place and working correctly.

Read the report

The script added can be triggered manually by going directly to the URL and it will run the script and publish the report that will look like something similar to

{
    message: "WARNING: No new URL found",
    report: {
        time: 1606836659,
        successfulTime: 1606842343,
        error: false,
        requests: [ ]
    }
}

The message visible can be different depending on the work done by the script here to summarise all different status.

The format is in JSON

• message: will show the result action done for the manual run just reported
• report: contains the latest report that runs
• report.time: Timestamp when the latest report performed.
• report.successfulTime: The timestamp of last time where the report was successful 
• error: if there script run in any error
• requests: the requested information performed to the SikTide API

How to read the message

Temporarily Disable the Script

It is possible to temporarily disable the script, if needed. In this case both a manual trigger or the scheduled one will not perform any action and no request will be sent to SilkTide.

1. Go to Site Structure and edit Home section
2. In the Advanced Section Information, check "Disable Silktide Integration"
3. Publish Section

This will disable the script and not allows it run until the flag is unchecked again and its change is not published out.

Change the config file

Another thing that maybe it is needed is to change the setting of the config.json file. To change this:

1. Go to Assets > Content Type
2. Edit the Section Metadata content type
3. Go to Content Layouts tab
4. Select text/silktide-config-json
5. In Content Layout Code it is possible to change the following variables
   1. silkTideApiURL = 'https://api.silktide.com/cms/update' (the SilkTide API Endpoint)
   2. silkTideApiKey = '<!-- INSERT SILKTIDE TOKEN →' (the SilkTide API Token) 
   3. silkTideDeferSec = 1800 (the number of seconds to wait before SilkTide will retest the URLs sent)
   4. bulkLimit = 1000 (the number of URLs sent in batches)
   5. delayInSeconds = 0 (the number of seconds to wait before each HTTP Request batches)
   6. basePage = 'index.php' (the basename to remove used as channel default)
6. Once the variables are changed Save the content layout.
7. And perform a Publish section of the Home to ensure that the new settings are applied
8. Please ensure to manually trigger the script to ensure that the change took place

Add the "Edit in CMS" button (silktide-cms meta tag)

The steps above send Silktide the pages that changed in each publish. Silktide can also show a button that opens the page in Terminalfour for editing. To show it, Silktide reads a silktide-cms meta tag from each page. The tag's content is a Base64-encoded JSON object with two keys: cms and editorUrl. editorUrl is the link to the page's section in Terminalfour.

The tag is generated by a Custom Handlebars Helper while the page is built, so it works on pages that publish as HTML or PHP. The page layout's file extension doesn't need to change.

Create the Custom Helper

1. Find the ID of the section where Custom Helpers are stored by running this query in the query handler:

   SELECT * FROM config_option WHERE config_key='handlebars.helpersSectionId'

2. Open that section in Site Structure. To do this, open any section and change the section ID at the end of the URL, e.g. https://[your-instance]/terminalfour/page/section#edit/[SECTION_ID]
3. Add a new content item with the following values:
   1. Name: silktideCmsTag
   2. Function Code:

      function (context, options) {
        var B64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
      
        function utf8Bytes(str) {
          var enc = encodeURIComponent(str), bytes = [];
          for (var i = 0; i < enc.length; i++) {
            if (enc.charAt(i) === "%") {
              bytes.push(parseInt(enc.substr(i + 1, 2), 16));
              i += 2;
            } else {
              bytes.push(enc.charCodeAt(i));
            }
          }
          return bytes;
        }
      
        function base64(bytes) {
          var out = "";
          for (var i = 0; i < bytes.length; i += 3) {
            var b0 = bytes[i], b1 = bytes[i + 1] || 0, b2 = bytes[i + 2] || 0;
            out += B64.charAt(b0 >> 2);
            out += B64.charAt(((b0 & 3) << 4) | (b1 >> 4));
            out += (i + 1 < bytes.length) ? B64.charAt(((b1 & 15) << 2) | (b2 >> 6)) : "=";
            out += (i + 2 < bytes.length) ? B64.charAt(b2 & 63) : "=";
          }
          return out;
        }
      
        try {
          var baseUrl = options.hash("baseUrl");
          if (!baseUrl) {
            log.warn("silktideCmsTag: no baseUrl hash argument supplied");
            return "";
          }
      
          var sectionId = apis.getSection().getId();
          if (!sectionId || sectionId <= 0) {
            log.error("silktideCmsTag: current section could not be resolved");
            return "";
          }
      
          var json = JSON.stringify({
            cms: "TerminalFour",
            editorUrl: String(baseUrl) + sectionId + "/contents"
          });
      
          return base64(utf8Bytes(json));
        } catch (e) {
          log.error("silktideCmsTag: failed: " + e);
          return "";
        }
      }

4. Save and approve it.

Note: Use plain JavaScript in the helper. Calls to Java classes such as java.util.Base64 fail without an error message and the helper outputs nothing, which is why the Base64 encoding is written in JavaScript.

Add the content layout

1. Go to Assets > Content Types and select the Section Metadata content type (usually Section Meta Description).
2. Go to Content layouts and add a new content layout with the following values:
   1. Name: text/silktide-metatag
   2. File Extensions: Default
   3. Syntax type: HTML/XML
   4. Content layout processor: Handlebars Content
   5. Content Layout Code (replace [your-instance] with the address of your Terminalfour instance):

      <meta name="silktide-cms" content="{{{silktideCmsTag baseUrl="https://[your-instance]/terminalfour/page/section#edit/"}}}">

3. Save it.

Note: Use triple braces ({{{ }}}) so the Base64 value is output exactly as it is. With double braces, the = padding is output as =. Browsers still read the value correctly, but it's harder to check in the page source.

Create the navigation object

1. Go to Assets > Navigation > Add new navigation > Related content.
2. Fill in the following values:
   1. Name: Silktide Integration: get Meta Tag
   2. Description: Returns text/silktide-metatag from Section Meta Description
   3. Primary Group: Global
   4. Options: [x] Show pending content in preview
   5. Output title: leave blank
   6. Fetch Method: Use current
   7. Content layout: Use alternate content layout
   8. Alternate content layout: text/silktide-metatag
   9. Before HTML: leave blank
   10. After HTML: leave blank
3. Save changes.
4. Build the t4 tag for this navigation object.

Note: The fetch method must be Use current, not Use section, so that each page gets the link to its own section. Leave Cache output unchecked for the same reason.

Add the tag to the page layouts

1. Go to Assets > Page Layouts and open the page layout you want to update.
2. In the header code, add the navigation object's t4 tag inside the <head>:

   <t4 type="navigation" name="Silktide Integration: get Meta Tag" id="[NAVIGATION_ID]"/>

3. Save changes.
4. Repeat for each page layout. We suggest testing on one page layout first.

Check the output

1. Preview or publish a page that uses the page layout and open the page source. It should contain a tag like this:

   <meta name="silktide-cms" content="eyJjbXMiOiJUZXJtaW5hbEZvdXIiLCJlZGl0b3JVcmwiOiJodHRwczovL...">

2. To check the value, decode the Base64. It should look like this, with the ID of the page's section:

   {"cms":"TerminalFour","editorUrl":"https://[your-instance]/terminalfour/page/section#edit/1234/contents"}

3. Publish the pages. The button appears in Silktide after it re-crawls them.

If the tag is missing or its content is empty, check the publish log for messages that start with silktideCmsTag:.