Thursday, April 21, 2016

[Analytics Dashboard] How to Create a Gadget Using External Data Set.


In this blog post I am guiding you to create a dashboard gadget using an external database.

As the example gadget I implemented a graph to display the blogger audience count by month.
Here I create a MySQL database called test. And I create a sample database table as follows.



Here I am using WSO2 DAS 3.0.0 version for the implementation. First you need to place the MySQL JDBC driver in  following path.

<DAS Home Path>/repository/Components/lib

Then you need to create an API to access the external database. The API should place in the
<DAS Home Path>/repository/deployment/server/jaggeryapps/portal/controllers/apis folder.
The API should implement using JAGGERY [1].


Here is my sample API code.

testAPI.jag

       
<%
//make the db connection 
var db = new Database("jdbc:mysql://localhost:3306/UnifiedDashboards", "root", "");
//function to get data from the db
function bloggerData() {
    //db query to get month and the pageviews data set from db
    var query_blogger_data = "SELECT * FROM test;";
    var bloggerDataSet = db.query(query_blogger_data);     
    print(bloggerDataSet);
    return bloggerDataSet;
}
bloggerData();
db.close();
%>
         
       
 

This API returns the dataset from external db.

Here I make API call via POST method in javascript code. And then in the success function I convert the data set in to JSON format and bind the data set with the graph.The code snippet  is as follows.

$.post("/portal/controllers/apis/api/testAPI.jag", function (dataSet) {
var data=JSON.parse(dataSet);
dataBind(data);
});

Now you need to create the gadget to visualize these data. You can follow the steps in [2] to create the gadget.
You can download sample gadget [3] which I implemented. Here I used a third party library[4] to draw the graph.

Flow Chart Diagram of the Gadget.





 UI of the Gadget.


References.






Saturday, April 2, 2016

Retrieve Repository Details from Git Hub API Using WSO2 GitHub Connector.



In this blog post I'm going to explain how to retrieve all the repository details for specified organization from Git Hub API using WSO2 GitHub Connector[1].

For this task following WSO2 GitHub Connector method can be used. 

       
<github.listOrganizationRepositories configKey="GH">
<org>wso2</org>
<type>public</type>
<page>1<page>
</github.listOrganizationRepositories>                     
       
 

Parameters :


org : Organization name
type : Type of the repository. Eg: public,private,forks etc. Default option is All
page : Page number 

But GitHub API uses pagination[2]. Therefore using the above method you can retrieve data only for the given page number . In other words there is no direct API call found to retrieve all the repository details for specified organization at once. To overcome this I successfully implemented the following ESB proxy service. Using this proxy service you can retrieve all the repository details for specified organization.
First you need to download WSO2 GitHub Connector [1] and add it to WSO2 ESB [4]. Then create a custom proxy service in ESB .

Proxy

 

       
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="Demo"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <github.listOrganizationRepositories configKey="GH">
               <org>wso2</org>
         </github.listOrganizationRepositories>
         <property name="link"
                   expression="get-property('transport','Link')"
                   scope="default"
                   type="STRING"/>
         <script language="js">
                var link = mc.getProperty('link');
                var lastPage=1;
                var pageLink=link.split(",");
               
                for(var i in pageLink)
                {
                 if(pageLink[i].indexOf("last")!=-1){
                        
                         var result=pageLink[i].match(/page=(\d+)/);
                          lastPage=result[1];
                         break;
                 }
                }
                
                mc.setProperty("lastPage", lastPage);
                var pageNumberDataSet =new XML('<pages></pages>');
                var pageNumberChild;
                var num;
                
                for(var i=0 ;i < lastPage;i++)
                {
                      
                     num=i+1;
                     pageNumberChild= new XML('<pageNumber>' + num + '</pageNumber>');
                     pageNumberDataSet.prependChild(pageNumberChild);
                   
                }
              
                 mc.setProperty("pageNumberDataSet",pageNumberDataSet);
          </script>

         <property name="pageNumberDataSet"
                   expression="get-property('pageNumberDataSet')"
                   scope="default"
                   type="STRING"/>
         <payloadFactory media-type="xml">
            <format>
               <root xmlns="">$1</root>
            </format>
            <args>
               <arg evaluator="xml" expression="get-property('pageNumberDataSet')"/>
            </args>
         </payloadFactory>
         
         <iterate continueParent="true"
                  id="pageNumberIteratorInRepoList"
                  expression="//root/pages/pageNumber"
                  sequential="true">
            <target>
               <sequence>
                  <property name="pageNum"
                            expression="//pageNumber/text()"
                            scope="default"
                            type="STRING"/>
                  <github.listOrganizationRepositories configKey="GH">
                     <org>wso2</org>
                     <page>{$ctx:pageNum}</page>
                  </github.listOrganizationRepositories>
                  <log level="full"/>
               </sequence>
            </target>
         </iterate>
      </inSequence>
   </target>
   <description/>
</proxy>                      
       
 

Description 

 

       
<github.listOrganizationRepositories configKey="GH">
<org>wso2</org>
</github.listOrganizationRepositories>                     
       
 


From the above connector method I got the link for the next page and last page as shown below.

Link: <https://api.github.com/resource?page=2>; rel="next",
      <https://api.github.com/resource?page=5>; rel="last"

Then I assigned the link header to a property called 'link'. 
       
<property name="link"
expression="get-property('transport','Link')"
scope="default"
type="STRING"/>                    
       
 




Then I split the link header and got the last page number inside the  script mediator. And then create a XML payload using page numbers. XML payload format is as shown below.
       
<root>
<pages>
<pageNumber>1</pageNumber>
<pageNumber>2</pageNumber>
</pages>
</root>                   
       
 
Then I iterate through the above payload using Iterator mediator. To do this you need to use the XPath expression as //root/pages/pageNumber.

Then I used the following GitHub Connector method to retrieve the repository details inside the Iterator mediator. The following Connector method gives the repository details for the specified page number.
       
<github.listOrganizationRepositories configKey="GH">
<org>wso2</org>
<page>{$ctx:pageNum}</page>
</github.listOrganizationRepositories>                   
       
 
Here I used Local Entry called “GH” to store GitHub Init operation and then it used in proxy as the configKey.

You need to create a local entry with inline XML as shown below. You can use [5] to create local entry.
       
<github.init xmlns="http://ws.apache.org/ns/synapse">
<accessToken>xxxxxxxxxxxxxxxxxx</accessToken>
<apiUrl>https://api.github.com</apiUrl>
<mediaType>application/json</mediaType>
</github.init>                  
       
 

You can create an access token using your Git Hub account [3].




Tuesday, March 22, 2016

[ESB] Aggregate Responses with WSO2 Aggregate Mediator




Aggregate mediator can be used to merge several responses and create one response by matching the Xpaths.
In this blog post I'm going to demonstrate how to aggregates the response messages which were split by Iterator Mediator. 

The flowing proxy wrote to get the GitHub Contributors details for a specified repository.

 
The response message of this proxy is as shown below.


But the expected response message is as shown below.

 
Solution :

To get the expected response message  need to merge the sub messages in to one. We can do this using Aggregate Mediator.

To do this  need to add the Aggregate Mediator inside the out sequence as shown in below figure.

 

In the aggregate mediator we need to add the following parameters.

Aggregate Id : you can give any proper Id . This parameter is an optional thing

Aggregation Expression : This is the Xpath expression specifying based on which elements to aggregate.

Completion Max-messages : -1
Completion Min-messages : -1

There are maximum and minimum number of messages that can exists in aggregation . Here I used both cases as -1 because I need to aggregate all the messages.

EnclosingElementProperty : This is used to accumulate the aggregated messages inside single property.
Here I used this as “ROOT”. This property need to define in the proxy as follows.

<property name="ROOT" scope="default">
<root xmlns=""/>
</property>


Sunday, March 13, 2016

[ESB]Transform Message Content using Payload Factory Mediator in WSO2 ESB

Payload Factory Mediator 

 

 

This mediator can be used to transform/replace the content of a message in to desired format(XML/JSON/text).
Eg: Let's consider about a client server system.
Server sends the JSON format responses , but in the client side understand the XML format responses only. In this case we need to transform the JSON format responses in to XML format responses. We can do this using Payload Factory Mediator. 

How to transform a message JSON to XML format


Sample response in JSON format .


{
    "login": "peff",
    "id": 45925,
    "avatar_url":
"https://avatars.githubusercontent.com/u/45925?v=3",
    "gravatar_id": "",
    "url":
"https://api.github.com/users/peff",
    "html_url":
"https://github.com/peff",
    "followers_url":
"https://api.github.com/users/peff/followers",
    "following_url":
"https://api.github.com/users/peff/following{/other_user}",
    "gists_url":
"https://api.github.com/users/peff/gists{/gist_id}",
    "starred_url":
"https://api.github.com/users/peff/starred{/owner}{/repo}",
    "subscriptions_url":
"https://api.github.com/users/peff/subscriptions",
    "organizations_url":
"https://api.github.com/users/peff/orgs",
    "repos_url":
"https://api.github.com/users/peff/repos",
    "events_url":
"https://api.github.com/users/peff/events{/privacy}",
    "received_events_url":
"https://api.github.com/users/peff/received_events",
    "type": "User",
    "site_admin": true,
    "contributions": 2065
  }
Let's transform the above response in to following XML format .

<contributor>
<name>peff</name>
<contribution>2065</contribution>
</contributor>

You can do this by adding the following payloadFactory mediator .



<payloadFactory media-type="xml">
<format>
<contributor xmlns="">
<name>$1</name>
<contribution>$2</contribution>
</contributor>
</format>
<args>
<arg expression="//jsonElement/login/text()" evaluator="xml"/>
<arg expression="//jsonElement/contributions/text()" evaluator="xml"/>
</args>
</payloadFactory>

You can create a payloadFactory through the UI.

To do this first you need to add the payloadFactory mediator inside the insequence of the proxy service as shown below.



 

Then you can create the payload format and you can add the arguments using Xpath expressions as shown in below figure.


 

Sunday, February 21, 2016

[Analytic Dashboards] Inter Gadget Communication.



In this blog post I'm going to demonstrate you how to build a communication between two gadgets in a dashboard.

Inter gadget communication

Let's think there are two gadgets in a dashboard called Gadget-A and Gadget-B. Through inter gadget communication these two gadgets can talk to each other. In simply the Gadget-A can send a message to Gadget-B and then the Gadget-B can update according to that message.
This is based on publisher-subscriber pattern. In this pattern what happen is the publisher publish a message through a channel and the subscriber listen to this channel and fetch the message and get updated according to the message.


 
As shown in above diagram a subscriber can subscribes to multiple channels and also publisher can publishes to multiple channels.

Implementation of Example Gadgets.

In this example I created two separate gadget called publisher-gadget and subscriber-gadget. In publisher gadget I created a simple drop down menu. From it we can select a name of a fruit.In subscriber gadget I created a view to display image of the fruit.

When we select a fruit name in publisher gadget then the publisher notify the selected fruit name to subscriber gadget. Then the subscriber gadget fetch the fruit name and display the image of the fruit.


Publisher Gadget.

Create a custom gadget folder inside ,
<DAS_HOME>/repository/server/jaggeryapps/portal/store/carbon.super/gadget/ 

Here I named my custom gadget folder as "publisher-gadget" . And the folder structure of the publisher gadget is as shown below.





Gadget.json

{
"id": "Publisher",
"title": "Publisher",
"type": "gadget",
"thumbnail": "store://gadget/publisher-gadget/index.png",
"data": {
"url": "store://gadget/publisher-gadget/index.xml"
},
"notify": {
"select": {
"type": "fruits",
"description": "This notifies selected fruit name"
}
}
}

The highlighted part need to add when the gadget is a publisher. Publisher notify the message through the channel(The Channel name is "select" and you can give a type name as you like.Here the type is fruit) .
Index.xml

<ModulePrefs title="Publisher" height="250" description="Publisher-gadget">
<Require feature="pubsub-2"/>
</ModulePrefs>

when we implement inter gadget communication we need to import the feature pubsub-2.


List.js

This file includes the javascript code.

 //get selected option of the selector
        var selection = $("#fruitList option:selected").text();
        //publish the selected option
        gadgets.Hub.publish('select', {
            msg : selection
        });

We need to include the above highlighted code snippet to publish the message to the subscriber.


Subscriber Gadget

The folder structure of the gadget.


 
"img" folder contains the all the images which are used in this gadget and "js" folder contains the javascript files.
 
Gadget.json

{
   "id": "subscriber",
   "title": "subscriber",
   "type": "gadget",
   "thumbnail": "store://gadget/subscriber-gadget/index.png",
    "data": {
              "url": "store://gadget/subscriber-gadget/index.xml"
},
   "listen": {
   "state-selected": {
    "type": "fruits",
   "description": "Used to filter based on state"
   }
}

you need to add above highlighted code snippet to show this is the subscriber gadget. Here "state-selected" indicates the channel which the subscriber is listening and the "type" indicates the type of the data coming through this channel .

Index.xml

Similar to the publisher gadget we need to import the pubsub-2 feature in the subscriber gadget as shown in below code snippet.

<ModulePrefs title="subscriber" height="250" description="subscriber-gadget">
<Require feature="pubsub-2"/>
</ModulePrefs>


subscriber.js

gadgets.HubSettings.onConnect = function () {
gadgets.Hub
          .subscribe(
                  'state-selected',
                           function (topic, data, subscriberData) {

    });
};

using the above code snippet you can fetch the data in subscriber gadget.

Where “topic” keeps the data about the publisher,“data” keeps the message published by the publisher and “subscriberData” keeps the data about the subscriber.

You can download the source code of two gadgets from here.

After creating these two gadget you need to add them to a dashboard. Then click on settings of the subscriber gadget and then you can see two buttons as shown below.


Click on this button and then you will be able to see the check box with publisher name as shown in below figure.

Please put the tick in both places. Now click on “view” button on your dashboard.
Now you can select a fruit name from publisher gadget and then you can see the image of the fruit in subscriber gadget.
The output of this example is as shown in below.



 



[Analytics Dashboards]How to Create a custom gadget in WSO2 Dashboard Designer .



We can host our own gadgets which are implemented using specific requirements in analytics dashboard. This blog post guide you to create and deploy a custom gadget in analytics dashboard.
To do this you need to run the WSO2 Data Analytic Server in your machine.

Step 1 : Create index.xml

You need to create a folder for your custom gadget inside 
<DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/carbon.super/gadget/
Here I created a folder called hello-world.
Add the following index.xml file inside your custom gadget folder(in my case inside hello-world folder). 
There is a specific format is available for index.xml file. You can get an idea about this format by reading[1].

index.xml

<?xml version="1.0" encoding="UTF-8" ?>
<Module>
  <ModulePrefs title="Demo Hello World gadget"
     author="LCG"
     height="230"
     scrolling="true">
  </ModulePrefs>
<Content type="html">
  <![CDATA[
     <html>
       <head>
       </head>
       <body>
            <a>Hello World</a>
       </body>
     </html>
   ]]>
</Content>
</Module>

 
Step 2 : Create Gadget.json

This file keeps the gadget definition.

{
    "id":"hello-world",
    "title":"HelloWorldGadget",
    "type":"gadget",
    "thumbnail":"store://gadget/hello-world/index.png",
    "data":{
             "url":"store://gadget/hello-world/index.xml"
      }

}

 
You need to give the proper id title for your gadget. You need to give the location of your gadget image in “thumbnail” and location of the index.xml file for the “url”.

Index.png
 
Folder structure of the gadget.


 
 
Now the custom gadget is created. Now you can add this gadget to your dashboard.To add this Click on “design” button in your dashboard.
Then click on following button.






Then you can see your created gadget as shown below. Click on gadget and Drag and drop to the grid to add the gadget to your dashboard. 



Click on “view” button in your dashboard to view your gadget. The view of the hello world gadget is as shown below.

Like wise you can add separate gadgets to one dashboard.

If you want you can edit the gadget. The dashboard designer provides following options.

Click on following icon we can drag and drop the gadget from one grid to another grid in layout.
 
 
Click on following icon we can  maximize the size of the grid.


 
Click on following icon  we can delete the gadget from dashboard.

Click on following icon we can change the setting of the gadget. If you click on this you can change the default settings of the gadget as shown in below figure.




 

[Analytics Dashboards] XML Structure of a gadget

We can create our own dashboards in WSO2 Dashboard Designer in DAS. These dashboards contains gadgets. There are some samples in WSO2 Dashboard Designer. But we can create our own custom gadgets and then we can add them to our own dashboard.
When we create a gadget we need to create a XML file. This XML file come up with a common format. In this blog post I am going to introduce about this common format.



As shown in above diagram this XML file is wrapped with <Module> tag and inside it there are three sections as Gadget Preferences , User Preferences and Content Section.

<?xml version="1.0" encoding="UTF-8" ?>

This is the first line of any gadget which is used to indicate the format of the document.
In this case it indicates this document  uses XML format with UTF-8 character encoding.

<module>

This node is use to wrap gadget preferences,user preferences and content section of the gadget.

<ModulePrefs>

This section is used to keep the meta data of the gadget such as title,author,description etc. And also we can define the dimensions of the gadget such as height,width etc under this section.

Ex :

<ModulePrefs title="Bar Chart"
author="WSO2 DAS Server"
height="230"
scrolling="true"
description="A generic Bar Chart gadget, that display patch-count Vs Month.">
</ModulePrefs>

<UserPrefs>

Under this section you can store your own configuration values and allow users to give inputs for the application. This part is an optional thing.

Ex :

<UserPref name="dataSource"
display_name="Data Source"
default_value="/portal/gadgets/bar-chart/datasource/dataFile4.jag">
</UserPref>

<Content>

Under this section you can write your own code of the gadget implementation such as HTML,Javascript,CSS etc.

Ex : If we use HTML then the content is defined as shown below.

<Content type="html">
<![CDATA[
<html>
<head>
</head>
<body >
</body>
</html>
]]>
</Content>