Node-RED: Lecture 6 – Intermediate flows

This lecture will build on the ideas you saw in lecture 5 and focus on examples that explore some of the key concepts from that lecture. The examples in this lecture are a little more complex than previous examples – mainly in the sense that the function nodes are more complex – but are still kept as clean and simple as possible.

This lecture will build on the ideas you saw in lecture 5 and focus on examples that explore some of the key concepts from that lecture. These include the ideas of context, messages and sub-flows. The examples in this lecture are a little more complex than previous examples – mainly in the sense that the function nodes are more complex – but are still kept as clean and simple as possible.

 

We use a cloud hosted version of Node-RED for these lectures called FRED. Sign up for a free account at FRED. Examples in the early lectures will work with other installations of Node-RED, later lectures use nodes you will need to install yourself if you don’t use FRED.

 

Examples

(Click to go directly to the example)

Example 6.1 Retrieving data from a web page

Example 6.2 Counting words in a string

Example 6.3 Using context to generate rolling averages

Example 6.4 Using the context element to share a function and make it accessible to all functions in the canvas

Example 6.5 Defining and using an iterator sub-flow

Example 6.6 Getting earthquake data from an external API and returning it as multiple messages

Example 6.7 Multiple inputs on a function node

Example 6.8 Letting a function node send multiple messages on a single output

Example 6.9 Creating a Blog Site With Node-RED

 

Example 6.1 Retrieving data from a web page

 

 

Let’s write a flow that scrapes the latest stock market indices from google at http://finance.yahoo.com/market-overview/ and format it using a function node. If you inspect the page shown in Figure 6.1 using the Chrome browser’s ‘Inspect Element’, you’ll find that the three indices have the same td.bld span tag. This tag can be used to retrieve all three indices with the html node. You’ll set up an inject node to trigger an http request to get the page, then the html node to get the elements with the td.bld span tag.

Figure 6.1 Stock market index page from Google.

 

Configure the http node and html nodes as shown in Figure 6.2

Figure 6.2 Configuration of http and html nodes to retrieve stock indices from Yahoo.

Then wire them up as shown in Figure 6.3, with a debug node to show the output.

 

Figure 6.3 Flow to retrieve the market indexes.

The output should contain the latest index values. When you click on the ‘trigger’ inject node, you will see the following in the debug pane:

Figure 6.4 Stock indices as array of strings.

That was easy, but what you really want is an array of JSON name:value pairs that uses the name to describe the values and shows the values as numbers:

Listing 6.1 Desired JSON output of stock indices

  1. [{
  2.     “index”:”S&P”,
  3.     “value”:2096.92
  4.  },{
  5.     “index”:”Dow”,
  6.     “value”:17511.34
  7.  },{
  8.     “index”:”Nasdaq”,
  9.     “value”:5059.35
  10.  }]

Let’s write a function node to format this the way you want it. Take a look at Listing 6.2. Line 1 takes the incoming stock prices and stores them in an array prices and then creates two new arrays, one to hold the output messages and the other to hold the textual name of the three different indices. Lines 5-8 loop through the incoming stock indices and for each one pushes a name:value pair to the output message array. At Line 8, you use the JavaScript replace function to remove all commas from the index values before parsing them as numbers. Let’s call this function format indices.

Listing 6.2 Function node code to format stock indices

  1. msg.headers = msg.originalHeaders;
  2. var prices = msg.payload;
  3. var newPayload = [];
  4. var priceIndex = [‘S&P’,’Dow’,’Nasdaq’];
  5. for ( var i=0; i<prices.length; i++) {
  6.     newPayload.push({
  7.     index:priceIndex[i],
  8.     value:Number(prices[i].replace(/,/g,”))
  9. });
  10. }
  11. msg.payload = newPayload;
  12. return msg;

Now wire this into your flow as shown in Figure 6.5.

Figure 6.5 Complete market index flow.

Once you deploy and test, your debug console should output something similar to Figure 6.6. The data now looks good and you can use this format for downstream nodes in your flow.

Figure 6.6 Debug output from new flow.

Example 6.2 Counting words in a string

Next, let’s write a more complex function node that receives some text in a message payload, then outputs multiple messages containing all individual words and the number of times each word was used.

Listing 6.3 Word count function

  1. var outputMsgs = [];
  2. var wordMap = {};
  3. var sentence = msg.payload.replace(/[.,-\/#!$%\^&\*;:{}=\-_`~()]/g,””);
  4. sentence = sentence.replace(/\s{2,}/g,” “);
  5. var words = sentence.split(” “);
  6. for (var i = 0; i < words.length; i++) {
  7.   var lowerCaseWord = words[i].toLowerCase();
  8.   if (!wordMap[lowerCaseWord]) {
  9.     wordMap[lowerCaseWord] = 1;
  10.   } else {
  11.     wordMap[lowerCaseWord] = wordMap[lowerCaseWord] + 1;
  12.   }
  13. }
  14. for (var prop in wordMap) {
  15.   if( wordMap.hasOwnProperty( prop ) ) {
  16.     outputMsgs.push({payload:{word:prop,count:wordMap[prop]}});
  17.   }
  18. }
  19. return [outputMsgs];

In Listing 6.3, the list of output messages and an object to hold the word counts (Lines 1 and 2) is declared. In lines 4 and 5 the payload is ‘cleaned’, removing punctuation and extra spaces using regular expressions. Regular expressions are a very useful tool for text processing; you can learn more about regular expressions by looking at the Mozilla Developers pages for JavaScript (here).  Line 7 splits the text into multiple words, then iterates through the words, creating a mapping of the lowercase version of each word to word count called wordMap in Lines 8-15.  Lines 16 to 18 split the wordMap into multiple messages in the outputMsgs array[1].   Finally, in line 21, an array of this array of messages is returned, sending them all to the first output port one at a time[2].

Let’s wire this up and see it run with some example text.  First, create a function node, and copy in the code above. Call it word count. Then add an inject node and add the following text:

Figure 6.7 Inject node to test word count function node.

Add a debug node and wire up the flow as follows:

Figure 6.8 Test flow for word count function node.

When you click on the inject node, you should see the list of word counts in the debug pane:

Figure 6.9 Debug output from word count function node.

Example 6.3 Using context to generate rolling averages

A special module called context,used to store data between function invocations, is available to function nodes. This can be useful when the function needs to retain state to do its processing. For example, it may be necessary to compute the average value of a sensor’s data readings over a period of time. Listing 6.4 computes a rolling average of the values received over the last 5 seconds, adding the ‘average’ field to the payload when more than 5 seconds have elapsed between received messages.

Listing 6.4 Rolling average function using context

  1. var currentTime = new Date().getTime();
  2. if (!context.lastTime) {
  3.     context.lastTime = currentTime;
  4.     context.sum = msg.payload.value;
  5.     context.count = 1;
  6. }
  7. if (currentTime-context.lastTime > 5000) {
  8.     // calculate average for previous messages
  9.     msg.payload.average = context.sum/context.count;
  10.     // start tracking average again
  11.     context.sum = msg.payload.value;
  12.     context.count = 1;
  13.     context.lastTime = currentTime;
  14. } else {
  15.     context.sum += msg.payload.value;
  16.     context.count +=1;
  17. }
  18. return msg;

Looking at listing 6.4, you first get the current time (line 1).  If there is no lastTime stored in context, you then save the currentTime, reset the sum and count context variables (lines 3-7).

If the currentTime is 5 seconds (5000 milliseconds)  greater than the last time (line 9), you calculate the average of the last values received and include that with the message payload (lines 11-13).  You then reset the sum, count and set lastTime to the currentTime to begin counting and summing up again.

If 5 seconds haven’t elapsed since the last message was received, the counts and sums need to be updated (lines 16-19). You then output the message which includes the latest value and the average if it was calculated for the interval (line 20).

To test this, let’s write another function node, that you’ll call ramp and that also uses context to generate values from 0 to 9 in sequence, as shown in Listing 6.5.

Listing 6.5 Test function to generate sequence of values using context

  1. if (!context.value) {
  2.   context.value = 0;
  3. }
  4. msg.payload = {
  5.   value:context.value
  6. }
  7. context.value +=1;
  8. if (context.value > 9) {
  9.   context.value = 0;
  10. }
  11. return msg;

Let’s configure an inject node to send data every second, as shown in Figure 6.10.

Figure 6.10 Inject node configured to inject a blank payload every second.

Then wire these up as shown in Figure 6.11.

Figure 6.11 Test flow for rolling average function.

The output should look like the one in Figure 6.12.

Figure 6.12 Debug console output showing rolling average.

Example 6.4 Using the context element to share a function and make it accessible to all functions in the canvas

This example shows how to use the context object and its global element to share data across function nodes. The example above used this to store a numerical value. However, one of the great things about JavaScript is that you can assign functions to objects. This allows you to share methods across your canvas without the need to redefine them in each consecutively used function node. The contrived example below simplifies this process for better understanding.

First, connect an inject node, two function nodes and two debug nodes like this:

 

Figure 6.13 Wire up functions for a context example

Let’s edit the first function node and add this code. Set the global element of the context object to have a new element, “hello”, which is an anonymous function that returns the string “Hello There”. Then set the message payload of this function to the return value of this global function. This will print out “Hello There”.

Figure 6.14 Define an anonymous function as a new context element

Now edit the second function node by setting up the message payload of this second function to concatenate the output of the ‘hello’ element (anonymous function) in the global element of the context object with the word “World”:

Figure 6.15 Use the anonymous function defined in another function node

Pressing the button on the inject node shows the output of each of the function nodes:

Figure 6.16 Output of context function sharing example

Despite being very simple, this example shows how easy is to use the context element to set not only variables that contain data, but also functions that can be shared and accessed by several functions in your canvas.

Two important aspects to consider are: firstly, the function setting the global element of the context object must come BEFORE any other function wishing to use that variable or function in the flow. Secondly, in the current version of Node-RED (1.11.0), the context object remains in memory after re-deployment until the Node-RED instance is rebooted. So even after deleting the function nodes that set an elements in the global element, these elements will still be accessible until set to null.

Example 6.5 Defining and using an iterator sub-flow

As mentioned in lecture 5, sub-flows can be used to package up functions into nodes in your node pallette. In this example, you’ll create a sub-flow that processes an array of values in a message payload and outputs a new processed array[3].  An upstream node will supply a message with a payload containing an array of values to be processed. The first output of the iterate node will output each of the input message array payload messages , contained in a message payload in order. The second output will output a message containing a payload with a new array containing processed values. To process data, you can wire up the sub-flow as shown in Figure 6.17, connecting a function node to provide some test data and process each element in the array.

Figure 6.17  Example sub-flow to iterate through list of values in an array payload.

Start by creating a new sub-flow tab. Drag in a function node and add the code from listing 6.6.

Listing 6.6 Iteration function node

  1.   var currentMsg = null, outMessage = null;
  2.   var iState = msg.iState;
  3.   if (!iState) {
  4.      // we received an initial message
  5.      // if the message is not an array, make it one
  6.      if( Object.prototype.toString.call(msg.payload) !== ‘[object Array]’ ) {
  7.            msg.payload = [msg.payload];
  8.      }
  9.      iState = {};
  10.      iState.index = 0;
  11.      iState.inArray = msg.payload;
  12.      iState.outArray = [];
  13.      msg.iState = iState
  14.    } else {
  15.     // save results from the last iteration
  16.      iState.outArray.push(msg.payload)
  17.  }
  18.   //If there are still objects left to iterate goto the next one in the original array
  19.   if (iState.index < iState.inArray.length) {
  20.     currentMsg = msg;
  21.     msg.payload = iState.inArray[iState.index];
  22.   } else {
  23.     currentMsg = null;
  24.     outMessage = msg;
  25.     msg.payload = iState.outArray;
  26.     delete msg.iState;
  27. }
  28. iState.index ++;
  29. return [currentMsg, outMessage];

In lines 1 and 2, several variables are declared.   currentMessage, holds the current input message; outMessage holds the final output message containing our processed array payload and iState is used to make it easier to access the current state of the iteration from the msg.iState property.  While context could be used to manage the function’s state, it’s simpler to hold state in the message, since you may receive new arrays as input before completing the iteration of another array.

In line 4, you check to see if iState exists. If not, assume that the message is a new input message.  Check that the payload is an array, and if not, make it one in lines 7-9. Then generate the iteration state object that includes the current index, input array, output array, and add it to the msg in lines 10-14.

If iState does exist (16-18), you know it’s an iteration message generated by this function node and processed in the loop back.  Push it into the output array to send out when you’re done.

Next, see if you’re done or not. If you need to iterate again (index is less than the length of the input array), set currentMessage to the message and set the payload to thecurrent index (lines 20-22).  When finished iterating, set the currentMessage to null, the outMessage to the message, the payload of the message to the outputArray and delete the iState property that is no longer needed.

Then increment the index and output the messages to the output endpoints. During the iteration, currentMessage is checked to ensure it holds a value, otherwise (when null) the loop terminates.  At the end, outMessage will be non null and sent to the second output.

Now let’s hook up inputs and outputs to our sub-flow as shown in Figure 6.18.  Let’s name this subflow ‘Iterate’.

Figure 6.18 Iterate sub flow.

To test out the flow, generate an array from 1 to 5 as in Listing 6.7, and multiply each element in this array by 5.  To do this, create a new function node called [1,2,3,4,5] and wire it in as shown in Figure 6.17.

Listing 6.7. function node [1,2,3,4,5] from test flow in Figure 6.17

  1. msg.payload = [1,2,3,4,5];
  2. return msg;

Write a second function that is used in each iteration, as in Listing 6.8, calling it times 5, and wire it up to the new subflow output 1 and input.

Listing 6.8 multiply payload times 5

  1. msg.payload = msg.payload * 5;
  2. return msg;

Finally trigger the flow, using an inject node, and add a debug node. After clicking on the inject node, the debug window should display [5,10,15,20,25].

Now that this iteration function is packaged in a sub-flow you can use it in any of your flows by simply dragging it in from the node palette like any other node. It will iterate over any array of integer values and apply the function specified.

Example 6.6 Getting earthquake data from an external API and returning it as multiple messages

This example demonstrates how to get data from an external API and how to separate that data using a function node. We will use data from an external API that provides access to earthquake data which is made available by the US geological survey (USGS). (http://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php)

First, let’s wire the flow like this:

 

 

Figure 6.19 Flow to query USDG earthquake data

Let’s edit the http request to get data from the following url as shown below (Fig 6.20):

http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/significant_month.geojson

This data contains significant earthquakes from the last month. The JSON node after the request will allow you to parse the body of the response into an object usable by the following function.

Figure 6.20 Setting up a http request node to query USDG earthquake data

Now edit the function node and add the code shown in Fig 6.21:

This code will create an array (outputMsgs) containing an array of messages built from your data. The for loop goes through the received JSON response and creates a new message with a payload containing lat, lng, value, message and timestamp values (only a few values from the JSON object sent from the server). It will then push this new message object to the outputMsgs array.

It will create a new variable (msg2) with a new message object containing a payload element with the string “Second Output”.

Finally, the function will return an array containing two elements. The first element is an array of messages (outputMsgs); the second element is a single message (msg2). Let’s configure the function to have 2 outputs, to match our returned array.

Figure 6.21 Function node code to query USDG earthquake data

When you click the button on the inject node, you can see that the top output contains an array of messages parsed from the data. This corresponds to the outputMsgs array in the function (See Fig. 6.22).

Figure 6.22 First output from the flow that queries USDG earthquake data

The second output contains the single message (msg2):

 

Figure 6.23 Second output from the flow that queries USDG earthquake data

Example 6.7 Multiple inputs on a function node

Function nodes in Node-RED were designed to process messages as single entities. However, in some cases your functions might depend on two separate data sources. There are many ways to handle these cases in Node RED. The following approach uses the context object in Node-RED and topics to let a function wait for several messages to arrive in order to return. You saw how to set-up and use context data in Example 5.3 in the previous lecture.

Let’s start by connecting two inject nodes, a function node and a debug node like this (Fig 6.24):

Figure 6.24 Setting up a flow to explore multiple inputs to a function node

Let’s edit the function node and add the following code (see Fig 6.25). This code will use the context object in Node-RED and add a data element.

Line 1 initializes the context object. Then the switch statement at line 2 looks for the topic field in the message. It uses this to set the task1 or task2 field of the context.data object. Any other message topic is ignored. Line 16 then checks to see if the function has received messages of both topic types (task1 and task2). If not, the function returns a null message and goes back to wait for another message. Otherwise, line 17 calculates the ratio and outputs it as a message.

Figure 6.25 Function node code to wait for all input before proceeding

Let’s configure the first inject node to return a string payload of “3”, with a topic of “task1”

Figure 6.26 Configuring an inject node with data and a topic used by a context object

You should configure the second input node to return a payload string of “6”, with a topic of “task2” (not shown but similar to Fig 6.26)

You can then deploy the flow. Click on the left button of the “task1:3” inject node. You will see a success message indicating that the string has successfully been injected, but you will not see anything in the debug tab. Click on the left button of the “task2:6” inject node. You will see a success message and the debug tab will show the ratio as expected:

 

Figure 6.27 Exercising the multiple input flow of Example 6.7

Example 6.8 Letting a function node send multiple messages on a single output

Example 5.2 in Lecture 5 demonstrated how to set up and send messages on multiple output nodes. This example shows how to send multiple message, but on the same output, from a single function node. One common scenario is a function node batch processing some data, with the following nodes wanting to have processed data as soon as it becomes available. Another common scenario is the creation of nodes that in turn create counters or pulses to actuate relays.

The following example is based on a flow by Node-RED contributor dceejay. Tthe original flow can be found at http://flows.nodered.org/flow/5c4c0f5a08d4e91ea14d

First, wire together an inject, function, delay and debug nodes:

Figure 6.28 A basic flow to explore multiple output messages

Edit the function node and add the following code (See Fig. 6.29). This code is quite simple; it contains a loop that calls “node.send” 5 times. node.send was introduced in Lecture 5 at the end of example 5.2. It allows a function node to output messages to its output independently of its return value. Note how the function itself returns null. The expected output of this function will be a series of messages with the payloads: 0,1,2,3 and 4.

Figure 6.29 Code to send multiple messages using the node.send() command

To make the example ressemble a counter more closely, you can edit the delay node and configure it to limit the rate of messages to 1 per second, as shown in Fig 6.30.

Figure 6.30 Configure the delay node to limit its output rate to 1 message per second

The flow can now be deployed. Press the button left of the inject node and inspect the debug tab (See Fig 6.31) and you will see the messages arriving every second:

 

Figure 6.31 Output from Example 6.8

 

Example 6.9 Creating a Blog Site With Node-RED

This final example will show you how to build a micro blog service with only a few nodes in Node RED. You’ll be using the MongoDB node as a storage for posts, http nodes to provide end points for the service and the html node to format the micro-blog web page.

We will use a free cloud service called “mongolab”. It allows you to create “sandbox” databases you can use to prototype your applications. Head over to https://mongolab.com. Register for an account. You will then be able to create a database. Click on “Create New” in your home dashboard.

Figure 6.32 Creating a free MongoDB at mongolab

Fill out the form to create a new MongoDB deployment. Select Amazon, Single-node and select the FREE size of 500mb.

Figure 6.33 Choosing the free MongoDB

Scroll down and name your database. We are naming it “mycontent”. Now click on Create new MongoDB deployment:

Figure 6.34 Name the free MongoDB

MongoDB allows you to create “collections” for each database. They are analogous to “tables” in relational databases, each “collection” holds “documents” analogue to “records” in relational databases. In order to begin using your database you need to create a new collection. Select your newly created database in your dashboard and add a new collection. We will name it “posts”.

Figure 6.35 Adding a collection to the  MongoDB

Finally, you will need a user to connect to that database. Add a new database user. We will name it “freduser” but you should use a name that makes sense to you.

Figure 6.36 Adding a user to your MongoDB

Finally, note how this page also gives you important information on how to connect to your MongoDB database. We will need the URI (in our case ds037234.mongolab.com), the port (37234 in our case) and your newly created user and password.

Figure 6.37 Note down the connection info for your new MongoDB

Now, head over to Node-RED and wire up a http-in, mongodb-in, a template, and an http-out node as shown in Fig. 6.38.

Figure 6.38 The basic flow for the micro-blog

Edit the http-in node to accept GET request on the URL “/public/posts” (Fig 6.39)

Figure 6.39 Setting up the http node to accept requests on /public/posts

We will now configure our mongodb in node. Double click on the node and create a new server connection. Here is where you will use the information from your mongolab instance:

Figure 6.40 Configuring the mongodb node using the MongoDB account info

Click on Create/Update and configure your node to use the “posts” collection you created. Configure it to do a “find” operation, which will find all documents in that collection. We will now name it “find posts”.

Figure 6.41 Configuring the mongodb node with the collection and the find operation

Once you have the http node and the mongodb nodes configured you can edit the html template node to handle the data returned by the mongodb node (Listing. 6.9):

 

Listing 6.9 Formatting the blog posts

And add the following code:

  1. <!DOCTYPE html>
  2. <html lang=”en”>
  3.   <head>
  4.   </head>
  5.   <body>
  6.     <h1>My Micro Blog</h1>
  7.     <form action=”posts” method=”post”>
  8.         <label for=”title”>Title</label>
  9.         <input type=”text” class=”form-control” name=”title” placeholder=”Title…” />
  10.         <label for=”post”>Post Content</label>
  11.         <input type=”text” class=”form-control” name=”post” placeholder=”Say something…” />
  12.         <button type=”submit” class=”btn btn-default”>Submit</button>
  13.     </form>
  14.     <div>
  15.         {{#payload}}
  16.         <div class=”post”>
  17.             <h3>{{title}}</h3>
  18.             <p>{{post}}</p>
  19.         </div>
  20.         {{/payload}}
  21.     </div>
  22.   </body>
  23. </html>

This code, while long, is very simple. You will be using Bootstrap to style your website (http://getbootstrap.com/) and provide a responsive layout. If you now visit http://{your user name}.fred.senstecnic.com/api/public/posts you will be able to see your new blog site (Fig 6.42). However there is no content yet.

 

Figure 6.42 Output from the micro-blog html template node

 

 

So far, you’ve created the flow needed to display posts. Now you need to create the other side of the blog service, a flow to create posts and save them to your MongoDB collection. Connect an http-in, a function, mongodb-out, and http-out nodes as shown in the bottom part of Fig 6.43.

 

Figure 6.43 Setting up a flow to allow you to post blog entries and store them in MongoDB

Edit the http-in node to accept PUT requests at /public/posts (Fig 6.44)

Figure 6.44 Setting up the end point for posts for the micro blog service

Now let’s edit the function node and add the following code (listing 6.10). This builds a simple http msg.header which your flow will pass to the http node.

 

Listing 6.10: Building a http request for the micro-blog service

  1. msg.headers = {
  2.     “Location” : “https://{your username}.fred.sensetecnic.com/api/public/posts”
  3. };
  4. msg.statusCode = 302;
  5. return msg;

 

Note that at line 2 you set the address to be the end point for your service. This will use your user name in place of ‘guides’.

Finally, you will need to edit the mongodb-out node and select your previously configured server. Configure it to use the collection “posts” and an “insert” operation. Make sure to check “only store msg.payload object”, this will make sure we only store the data in the message payload, not the whole message object (fig 6.45).

Figure 6.45 Configuring the mongodb node to store the incoming blog postings

That’s it! You can now go to your url and use your micro blog site where you’ll see something similar to Fig 6.46.

 

Figure 6.46 Screen show of the simple micro-blog web page

 

 

Summary

In this lecture you have seen a number of more complex flows that build on the nodes and techniques introduced in earlier lectures. In particular, you’ve seen how to scrape data from a commercial web site and query web-based APIs, how to use context and message in more complex ways and how to build a simple web service to support a micro-blog. The next lecture will conclude our intermediate section (lectures 4-7) by introducing how to install Node-RED on devices such as the Raspberry Pi. It will show some examples that you can’t run on the cloud-based FRED service (because they require access to the underlying hardware), but which do run on the Pi, and it will discuss some of the issues with flows in devices such as the Pi.

 


About Sense TecnicSense Tecnic Systems Inc have been building IoT applications and services since 2010. We provide these lectures and FRED, cloud hosted Node-RED as a service to the community. We also offer a commercial version to our customers, as well as professional services. Learn more.


© Lea, Blackstock, Calderon

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

 

 

 


[1] The check for hasOwnProperty() in line 17 is good practice when iterating through the properties of an object using the JavaScript in operator. It ensures you’re getting only properties you’ve added yourselve and none that you’ve (unexpectedly) inherited from the Object base class.

[2] Note that if you returned outputMsgs directly, the function node would attempt to send each message to a different output.

[3] Based on forEach (array iteration & completion) subflow by dhartveld at https://gist.github.com/dhartveld/43501a1b424434de0ffb

Node-RED: Lecture 2 – Building your first flows

This lecture will introduce you to the Node-RED visual tool and get you started building your first flow. You’ll learn how to create simple flows, use debug nodes to track messages through the flows and how to use the function node to write simple JavaScript code that tailors the node to your specific needs.

This lecture will introduce you to the Node-RED visual tool and get you started building your first flow. You’ll learn how to create simple flows, use debug nodes to track messages through the flows and how to use the function node to write simple JavaScript code that tailors the node to your specific needs.

To get you started, you’ll be using a free cloud service that provides you with a preconfigured Node-RED setup and a wide range of built-in nodes. Later lectures will explain how to set-up Node-RED for yourself on devices such as a Raspberry PI.

At the end of this lecture you’ll be able to create your own simple flows, you’ll know your way around the Node-RED UI and you’ll be familiar with some of the basic built-in nodes that Node-RED offers.

FRED – Front end for Node-RED

FRED is a cloud service that we’ve created that allows you to sign up and use Node-RED without the hassle of downloading and installing on a device such as a PI. It makes it easy to get started with Node-RED by managing an instance of Node-RED for you, keeping the instance up to date with the latest code releases and applying bug fixes or patches. Also included is a set of extra Nodes that are contributed by the community, which add to the standard basic set you get when you install Node-RED yourself. However, FRED is just a wrapper for Node-RED, so anything you write in FRED will work on a Node-RED instance running on a Pi or a Beagleboard.

To begin our examples, make sure you have created your own Node-RED instance in the cloud by signing up for a free account at http://fred.sensetecnic.com.

After registering, make sure to activate your account via your email. You will not be able to log in until you validate your account.

The Node-RED UI

Once you have logged in to your FRED account, you’ll see the standard Node-RED UI which consists of three main panes, as shown in Fig 2.1

Screen Shot 2015-10-22 at 4.53.21 PM.png

Figure 2.1 The Node-RED UI  – showing the node palette (left), flow canvas (centre) and output pane (right)

The main pane is the flow creation workspace in the middle. This is where you drag and drop nodes and connect them with wires. Along the top of the workspace pane is a set of tabs. Each tab opens a previously created workspace and shows any flows created using that workspace.

On the left is the node pane that contains all the built-in nodes that your instance of Node-RED supports. In the example above, you are seeing the FRED node set which contains a large selection that has been added to the basic set that comes with Node-RED. In later lectures you will learn about these nodes and use them to develop new flows. As you can see, nodes are grouped into categories. Opening up a category shows the individual nodes.

On the right-hand side is the output pane that can be toggled between the info and debug tabs. When info is selected, documentation for the selected node is shown there.  When debug is selected, it will display the output of debug nodes, errors and warnings.

Above these three main panes is the usual toolbar, and on the right-hand side are three widgets, a deploy button, a user info icon and a pulldown menu for admin and control. You’ll look in more detail at the pulldown later in these lectures. The user info icon allows you to return to the FRED front page with links to tutorials, your account information, status and other information as well as to log out of the FRED service.

The Deploy button is used when a flow has been constructed and causes the flow to be deployed onto the Node-RED system and executed. You’ll be introduced to the details of what’s actually happening under the covers in lecture 5 and beyond. For now just treat the Deploy button as the way to get your flow running.

A quick tour of Node-RED nodes and messages

As you saw in lecture 1, Node-RED allows you to wire together nodes to create flows which carry out your programming task. Messages pass between nodes, moving from input nodes through processing nodes to output nodes. Let’s take a brief look at nodes, flows and messages.

There are three main  types of nodes:

  1. Input Nodes (e.g. inject)
  2. Output Nodes (e.g. debug)
  3. Processing Nodes (e.g. function)

basic-fred-intro-02

Figure 2.2 The main types of nodes: input, output and processing

Input nodes allow you to input data into a Node-RED application or “flow”. They have at least one output endpoint represented by the small grey square only on their right side. You use input nodes to connect data from other services, for example the  Twitter, Google, serial, websockets or tcp nodes, or to manually input data into a flow using the inject node.

Output nodes allow you to send data outside of a Node-RED flow. They have a single input endpoint on their left side. You use output nodes to send data to other services, for example via Twitter, tcp, serial or email nodes, or to use the debug node to output to the debug pane.

Processing nodes allow you to process data. They have an input endpoint and one or more output endpoints. They allow you to transform the data type (e.g. json, csv, xml) nodes, use the data to trigger a message (e.g. trigger, delay) nodes and to write custom code that uses the data received (e.g. function node).

Note that some nodes, like the inject and debug messages, have a button that allows you to actuate a node (in the case of the inject node) or to enable and disable a node (in the case of the debug node).

Flows consist of multiple nodes wired together, with output tabs linked to input tabs of the next node in the flow. Messages flow along the nodes carrying data from node to node.

Node-RED nodes consume input messages and produce output messages. Messages are JavaScript objects that contain at least a “payload” parameter, like this:

Listing 2.1 A basic Node-RED message structure

  1. msg = {
  2.   payload:”message payload”
  3. };

Nodes consume and produce messages, usually using msg.payload as the main placeholder for the data they consume and produce. However, messages can be extended to contain other parameters. For example, to set the topic of a message and add a new parameter, location, you could create a new msg object as shown in listing 2.2.

Listing 2.2 A more complex Node-RED message structure

  1. msg = {
  2.   payload:”message payload”,
  3.   topic:”error”,
  4.   location:”somewhere in space and time”
  5. };

Let’s use this knowledge to create your first flow.

Example 2.1 Building your first flow: Hello World

Let’s go ahead and start building your first flow so that you can see how simple it is to use the Node-RED UI to build and deploy a flow.

Let’s start with the easiest flow possible, a node to inject some information into the flow, wired to a debug node to see the output from the flow as a debug message. Once you have that running, you’ll build it up to the full Hello World flow.

Since this is the first time you’ll be shown how to actually build a flow, let’s start slowly and explain each step with a screen-shot. Once you’ve created your first flow and seen how easy it is, you’ll mostly be shown a single picture of the final flow rather than all the screenshots.

Let’s start with the simplest Node, the comment Node. You’ll find this in the function section of the node palette. Drag and drop a comment node onto the flow workspace as shown in Fig 2.3.

 

Screen Shot 2015-07-31 at 2.03.20 PM.png

 

Figure 2.3: Using a comment node is a great way to add visible comments into flows

 

Once you’ve done that, take a look at the info pane on the right (remember to switch to info if the debug tab is selected). You’ll see a little info on the node, including the node name, a unique ID, and a properties field with a description of the node. For the comment node there’s not much to say; however, more sophisticated nodes have a lot more info.

Double-click on the comment node, and you’ll see a configuration window (Fig 2.4) You can give the comment node a name and add detailed text if you like.

 

Screen Shot 2015-07-31 at 12.16.33 PM.png

Figure 2.4 Give the comment a name and add any info you want in the text box

 

Ok, now let’s add the first node that actually does something, an inject node. The inject node is used to generate input into a flow and is one of the first nodes in the node palette under input. If you drag and drop an inject node onto the flow workspace, and then look at the info tab, you’ll see the documentation for the inject node. Note that the name of the node on the workspace changes from inject to timestamp, because the default behaviour for the node is to inject a timestamp – the current time in milliseconds since January 1, 1970.

 

Screen Shot 2015-07-31 at 12.03.59 PM.png

Fig 2.5 The inject node allows you to insert events as messages, defaulting to a timestamp

 

You’ll also notice that the inject node (now named timestamp) has a blue dot top right and a grey square centre right. The blue dot indicates that the node hasn’t been deployed since it was last changed; the grey square is the output point for the node. This is where you attach ‘wires’ that route output message from the inject node to the next node in the flow.

To get a sense for the inject node and the whole flow deployment process, let’s add a debug node to see what happens, wire them together and then deploy the flow and test it.

Start by dragging a debug node from the node palette to the workspace. Again you can look at the info for the node.

Then you’ll wire the two nodes together. To do that, click on the grey output point for the inject node and, holding the mouse button down, drag towards the debug node. An orange wire appears, which you can then attach to the grey input point on the debug node.

Screen Shot 2015-07-31 at 12.40.55 PM.png

Figure 2.6 Wiring an inject node to a debug node

 

This is the simplest flow possible and will send the current timestamp to the debug node for display in the debug pane. Let’s try it!

Click the deploy button in the Node-RED window (top right). You’ll see a pop-up saying the flow has been successfully deployed. You will also notice that the blue dots on the nodes disappear, indicating there are no undeployed changes.

Now, before you try the flow, make sure the the debug tab is selected on the right pane. Then click on the left tab on the inject node and look at what appears in the debug pane.

 

Screen Shot 2015-07-31 at 12.48.44 PM.png

Figure 2.7 Debug output from your first flow – the time when you clicked the inject node.

 

As you can see, the inject node, when clicked, generates a timestamp (number of milliseconds since January 1, 1970), which is converted to a message and sent along the output wire, which is delivered to the debug node as an input message. The debug node’s default behaviour is to show any message it receives, which it does in the debug pane on the right.

Congratulations, you created and deployed your first flow!

Let’s now augment it a little to see what else we can do with this simple flow. Firstly, we’ll edit the inject node to deliver a text message rather than a timestamp. To do that, select the inject node in the flow and double-click it. You’ll see a configuration window like the one in Fig 2.8.

 

Screen Shot 2015-07-31 at 12.54.29 PM.png

Figure 2.8 Editing an inject node to send text instead of a timestamp

 

In the payload field, select string instead of timestamp and then type any string you like into the blank field below the payload field. As is traditional, let’s start with “Hello World – from my first NR flow!”

Once you’ve made the change, click ok to save the changes and take a look at the flow. You’ll see the blue dot has appeared on the inject node (which has also returned to being called inject rather than timestamp) to indicate that you have undeployed changes. Click the deploy button again to resolve that and then go ahead and click the tab on the inject node. If you look at the debug output you’ll see that instead of a timestamp, your text has been delivered as a message to the debug node, which displays it as usual.

Screen Shot 2015-07-31 at 1.01.40 PM.png

Figure 2.9 Sending a text message rather than a timestamp

 

As you can see, it’s very easy to wire up some simple nodes and get data to pass through your flow as messages. At this stage, you’ll probably be wondering a little about the messages that are flowing between nodes. The debug node can be used to examine them in more detail.

All messages in Node-RED have three default properties: the payload, which we are seeing above, a message topic, which is a user-defined string describing what the message is about (its topic, if you will) and an internal identifier. You can actually see this information if you edit the debug node configuration. Let’s do that and look at the output (Fig 2.10)

 

Screen Shot 2015-08-06 at 11.15.08 AM.png

 

Figure 2.10: setting the debug node to show the internals of a message

 

Select the debug node, double click and change the node output field to “Complete msg object”. If you then save this, deploy again and click the inject node, you will see in the debug pane a JSON structure that contains 3 fields: a “topic” which is currently blank, a “payload” which contains the string you set in the inject node and an internal ID field “_msgid”. Generally the internal message ID field isn’t used. However, both the topic and payload are used extensively when developing flows. You’ll see more of that later in the lecture series.

These fields are extensible, so you can define new message properties, for example, msg.location, which could be used to add the latitude and longitude values for the source of the message. Let’s take a look at that in the next example flow.

 

You can find the node-red description of this flow at:

https://github.com/SenseTecnic/nrguideflows/blob/master/lesson2/2-1-firstflow.json

Example 2.2 A second flow: weather alerts

In this example, which is similar to the one introduced in lecture 1, you’ll monitor the weather in your hometown and send yourself a tweet when the weather looks like it’s going to be good. You’ll use a built-in weather node – openweathermap – that retrieves the weather from openweathermap.org for the location you set it for. A simple function node will be used to check for ‘clear weather’ and a Twitter node will be used to send yourself a tweet when the weather looks good.

First you will need to get an API key at OpenWeatherMap. OpenWeatherMap offers a great service that provides detailed weather information for worldwide locations. Visit http://openweathermap.org/appid and follow the instructions as shown in the screenshots below.

 

 

You will need to sign up for an OpenWeatherAccount as shown below:

After signing up you will be redirected to your Home page where you will be able to access or re-generate your API Key. It should look something like this:

Now, let’s use that API key to build a weather alerts flow. Drag and drop a weather node from the left pane onto the workspace, as shown in Figure 2.11.

 

Figure 2.11. Drag and drop a weather node onto a blank workspace

 

If you select the info pane on the right, you can see a description of the openweathermap node with detailed information on how to configure and use it. Some interesting things to note:

  • It has a full JSON structure as its msg.payload, with quite a lot of weather detail, all presented as name:value pairs, e.g. wind speed and tempc
  • The node defines 3 new message properties, msg.location, msg.time and msg.data. As mentioned above, you are free to add properties to messages, and the openweathermap node has added these new properties to carry extra information associated with the message.

Let’s configure the node and look at the actual data structure it generates after it has queried your local weather. Start by double-clicking on the node and fill out the form with your location. Type your city and country in the fields. Add the API key you obtained from http://openweathermap.org/appid and click on “Ok” as shown in Fig. 2.12.

 

 

Figure 2.12. Set your city and country in the configuration form, use the API key you obtained in the step above.

 

Then drag and drop a debug node and wire it to the openweathermap node. Click on “Deploy” to see that the payload object from the openweathermap node in the debug pane.

 

Listing 2.3 The message payload for the openweathermap node is a JSON structure describing weather conditions, temperature, wind, cloud cover and sunrise times.

  1. {
  2.   “weather”: “Clear”,
  3.   “detail”: “sky is clear”,
  4.   “tempk”: 295.104,
  5.   “tempc”: 21.903999999999996,
  6.   “humidity”: 53,
  7.   “maxtemp”: 295.104,
  8.   “mintemp”: 295.104,
  9.   “windspeed”: 2.22,
  10.   “winddirection”: 273.007,
  11.   “location”: “Vancouver”,
  12.   “sunrise”: 1432038196,
  13.   “sunset”: 1432094081,
  14.   “clouds”: 8,
  15.   “description”: “Vancouver weather(49.25,-123.12) is Clear (sky is clear).”
  16. }

As you can see, the node delivers quite a lot of information about your local weather, all as the usual name:value pairs. For this example, you want to use the “weather” field which you’re going to test to see if it’s reported as “Clear”. If it’s “Clear” you’ll then send a tweet.

To program that logic, you’ll use a function node. You saw one of these in lecture 1, but without the details. Let’s see how they are used by dragging one onto the flow workspace and double clicking it to open up the node editor.

Figure 2.13. Add a function node and wire it to the openweathermap node.

Now double-click on the function node and type/copy this (Note if you use copy/paste – make sure you paste as plain text, wordpress sometimes inserts odd characters)

 

Listing 2.4 Function node “If Clear Weather”

  1. if (msg.payload.weather === “Clear”) {
  2.     msg.payload = “Clear skies ahead today!”
  3.     return msg;
  4. }
  5. return null;

Figure 2.14. Edit the function node adding the JavaScript shown in listing 2.4

Looking at Listing 2.4, you can see that this will parse the incoming message payload for the weather parameter and compare it to the string “Clear” (line 1). If it is equal, it will rewrite the message payload with your own string “Clear skies ahead today!” (line 2). Otherwise it will return a null message (line 5). This last bit is important because in Node-RED nodes ignore null messages.

You can do all sorts of things now, for example wire this message to an email node, or a pushbullet node. For this tutorial we will use the Twitter output node. Drag  a Twitter node onto the workspace, double-click and and fill out your Twitter account credentials as shown in Fig 2.15.

Figure 2.15. Set your Twitter account credentials into the node configuration

Once you’ve set up these credentials, Node-RED stores them and you can reuse them when you create another Twitter node in future flows.

Once you’ve wired up the flow, you can hit the deploy button and then watch your Twitter account to see the new tweet every time the weather is reported as clear.

Click on “Deploy”:

Figure 2.16. The full weather flow sending tweets if the weather report mentions clear skies.

Other useful UI features

One of the most useful features of the Node-RED UI is the ability to copy and paste nodes and even full flows using the standard cut’n’paste clipboard. Node-RED flows are simply coded as JSON strings and can be exported from a workspace, and imported into a workspace using the Node-RED pulldown menu in the top right of the Node-RED window.

Rather than building the flows in this example, or in fact anywhere in this lecture series, by hand yourself, you can simply copy them from our website and paste them into a workspace. After that all you need to do is configure them correctly, e.g. with credentials, and deploy them.

Let’s quickly show you how to do that.

Click on the link for this example which is at the end of the example below and look for the JSON code. Copy it to the clipboard using the usual CTL C or copy item in the edit menu.

Using the pulldown menu in the top right of the Node-RED window, select Import->Clipboard as shown in Fig 2.17

Screen Shot 2015-08-15 at 12.55.58 PM.png

Figure 2.17 Selecting the Import from Clipboard menu item in Node-RED

 

You will see a popup with an input field (Fig 2.18). Paste your clipboard into the input window and click OK.

 

Screen Shot 2015-08-15 at 12.56.27 PM.png

Figure 2.18 Pasting a flow into the import window of Node-RED

 

The new flow will appear in the current workspace and can be moved to a location and dropped. Once it’s in place, you can configure it as you would a flow you created yourself and deploy it.

 

You can find the node-red description of this flow at:

https://github.com/SenseTecnic/nrguideflows/blob/master/lesson2/2-2_weatheralerts.json

Summary

In this lecture you have created your first flows and seen how to wire together a set of basic nodes to achieve quite complex tasks. We’ve skipped over a few of the underlying details in order to get you going quickly, and we’ll return to those in lectures 3 and 4. However, so far you’ve seen how to use the Node-RED visual flow builder, the basic classes of input, output and processing nodes. You were given a brief overview of messages and how to use the function node to write your own JavaScript code to do simple processing on messages within a flow. In the next lecture, you’ll take a more in-depth look at the programming model of Node-RED and get a better understanding of the main programming elements and nodes and how to craft more complex flows using a variety of nodes.

 


About Sense Tecnic: Sense Tecnic Systems Inc have been building IoT applications and services since 2010. We provide these lectures, and FRED, cloud hosted Node-RED as a service to the community. We also offer a commercial version to our customers, as well as professional services. Learn more.


© Lea, Blackstock, Calderon

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.