Saturday 1 December 2012

Installation Of Hive In Ubuntu


Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL.

The hive-0.8.1 installation is done in below versions of Linux, Java and Hadoop respectively.

UBUNTU 12.04 LTS
JAVA 1.7.0_09
HADOOP 1.1.0

I have hduser as a dedicated hadoop system user. I had installed my Hadoop in /home/hduser/hadoop folder. Now I am going to install hive  in /home/hduser folder. Change the directory to the hduser and execute below commands.

Download the hive from below URL using wget.

Unzip the tar file.
sudo tar xzf  hive-0.8.1.tar.gz

Change the name to hive.
sudo mv hive-0.8.1 hive

Set the HIVE_HOME path.
I had downloaded the hive to the /home/hduser/hive folder
export HIVE_HOME=/home/hduser/hive
export PATH=$PATH:$HIVE_HOME/bin

Type hive in command line.

To list the tables use show tables hive command.

Installation Of Sqoop In Ubuntu


Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured data stores such as relational databases.

The Sqoop-1.4.0-INCUBATING installation is done in below versions of Linux, Java and Hadoop respectively.

UBUNTU 12.04 LTS
JAVA 1.7.0_09
HADOOP 1.1.0

I have hduser as a dedicated hadoop system user. I had installed my Hadoop in /home/hduser/hadoop folder. Now I am going to install sqoop in /home/hduser folder. Change the directory to the hduser and execute below commands.

Download the Sqoop from below URL using wget.

Unzip the tar file.
sudo tar xzf  sqoop-1.4.0-incubating.tar.gz

Change the name to Sqoop.
sudo mv  sqoop-1.4.0-incubating Sqoop

Set the SQOOP_HOME path.
I had downloaded the Sqoop to the /home/hduser/sqoop folder.
export SQOOP_HOME="/home/hduser/sqoop"
export PATH=$PATH:$SQOOP_HOME/bin

Test your installation by typing
sqoop help

Download the appropriate database driver and store it in lib folder /home/hduser/sqoop/lib.

Go to the ->sqoop -> bin and test examples…

Tuesday 14 August 2012

JQuery Ajax Maintaining State Between Multiple Requests:-


JQuery ajax function is used to call the server resources. While making the ajax request  probability of getting errors is more. In my scenario i am calling wcf REST service to get live cricket scores for multiple matches by sending matchId .If the call is success some how we can know the matchId ,but if error occurs there is no way to find the matchId of that request.To solve this issue we have to maintain state between multiple requests.
Using jquery ajax context property and object oriented concepts  we can  maintain state between multiple ajax requests.
In the below example i am having the wcf REST service which gives the live cricket scores,and javascript client which makes the ajax request.


    //Call the wcf service.
        function CallService(matchId) {

            //MatchId.
            this.matchId = matchId;

            //Make the ajax request.
            this.getScore = function () {

                //Ajax request.
                $.ajax({
                    //Context object to maintain state.
                    context: this,

                    //Url of wcf service.
                    url: 'http://localhost:6247/Service.svc/GetScore?matchId=' + this.matchId.toString(),

                    //Method type.
                    type: 'GET',

                    //Asynchornous call.
                    async: true,

                    //Success event.
                    success: function (res) {

                        //Disply score.
                        debugger;
                    },

                    //Error event.
                    error: function (err) {

                        //Get score again.
                        this.getScore();
                    }
                });
            }
        }

        //Create call service object.
        var obj = new CallService("M1");

        //Call getscore function.
        obj.getScore();

Here I didn’t give the code for wcf REST service which is out of scope,but it contains the simple GetScore function which takes the matchId as input and returns the live cricket score of that matchId.
Let me know, if you have any feedback. Mail me for source code. Enjoy reading my articles…
sekhartechblog@gmail.com


Saturday 28 July 2012

WCF IDispatchMessageFormatter


Wcf extentions gives more flexibility to programmers.There are many extension points in wcf  like IParameterInspector  ,IDispatchMessageInspector and IDispatchMessageFormatter etc…We can add these functionality’s at different levels at operation level ,contract level ,endpoint level and at service level. In wcf there are four interfaces like IServiceBehavior ,IEndpointBehavior ,IContractBehavior and IOperationBehavior which are used to add the above extension points.

Using IDispatchMessageFormatter we can send our custom Message object to the client.There are two methods SerializeReply and DeserializeRequest .The DeserializeRequest deserializes the message into array of parameters.SerializeReply serializes a reply message from a specified message version, array of parameters ,and a return value.

 I am having different clients like Android,WindowsPhone,IPhone,Blackberry and Web application.I have to send different propertys to different clients.To solve this I had used IDispatchMessageFormatter and Newtonsoft library .

From client side user sends the User-Agent in header .
In service side we had used that User-Agent header to serialize the message.

Create the attribute classes for all the clients.

    [AttributeUsage(AttributeTargets.All)]
    public class Android : Attribute
    {
    }

    [AttributeUsage(AttributeTargets.All)]
    public class WindowsPhone : Attribute
    {
    }

    [AttributeUsage(AttributeTargets.All)]
    public class IPhone : Attribute
    {
    }

    [AttributeUsage(AttributeTargets.All)]
    public class BlackBerry : Attribute
    {
    }

    [AttributeUsage(AttributeTargets.All)]
    public class Web : Attribute
    {
    }

The following class  contains the properties for different clients,those are set with specific attributes.
       
    [DataContract]
    public class Data
    {
        [DataMember]
        [Android]
        public string AndroidSpecific { get; set; }

        [DataMember]
        [WindowsPhone]
        public string WindowsPhoneSpecific { get; set; }

        [DataMember]
        [IPhone]
        public string IPhoneSpecific { get; set; }

        [DataMember]
        [BlackBerry]
        public string BlackBerrySpecific { get; set; }

        [DataMember]
        [Web]
        public string WebSpecific { get; set; }
    }

Now the following service contains the method to return the Data object.

    [ServiceContract]
    public interface IService
    {
        [OperationContract]
        [WebGet]
        Data Get();
    }

   public class Service : IService
   {
        public Data Get()
        {
            Data objData = new Data();

            objData.AndroidSpecific = "AndroidSpecific";
            objData.WindowsPhoneSpecific = "WindowsPhoneSpecific";
            objData.IPhoneSpecific = "IPhoneSpecific";
            objData.BlackBerrySpecific = "BlackBerrySpecific";
            objData.WebSpecific = "WebSpecific";

            return objData;
        }
 }

Here we are using the Newtonsoft library for serializing the object by watching the attributes.For this  we have to  use  the DefaultContratResolver’s  GetSerializableMembers method.In this method we can decide the serializable propertys.

public class Resolver : DefaultContractResolver
    {
        protected override List<MemberInfo> GetSerializableMembers(Type objectType)
        {
            //Get all the members.
            var allMembers = base.GetSerializableMembers(objectType);

            //User agent.
            string userAgent = string.Empty;

            //Get the user agent from client.
            userAgent = WebOperationContext.Current.IncomingRequest.Headers["User-Agent"];

            //The list of serializable members.
            List<MemberInfo> lstSerializableMembers = new List<MemberInfo>();

            //Iterate through all the members.
            foreach (var memberInfo in allMembers)
            {
                switch (userAgent)
                {
                    //For android.
                    case "android":
                        //Check whether the attribute exist or not.
                        if (memberInfo.GetCustomAttributes(typeof(Android), true).Length > 0)
                        {
                            //Add the property to list.
                            lstSerializableMembers.Add(memberInfo);
                        }
                        break;
                    //For iphone.
                    case "iphone":
                        //Check whether the attribute exist or not.
                        if (memberInfo.GetCustomAttributes(typeof(IPhone), true).Length > 0)
                        {
                            //Add the property to list.
                            lstSerializableMembers.Add(memberInfo);
                        }
                        break;
                    //For blackberry.
                    case "blackberry":
                        //Check whether the attribute exist or not.
                        if (memberInfo.GetCustomAttributes(typeof(BlackBerry), true).Length > 0)
                        {
                            //Add the property to list.
                            lstSerializableMembers.Add(memberInfo);
                        }
                        break;
                    //For windowsphone.
                    case "windowsphone":
                        //Check whether the attribute exist or not.
                        if (memberInfo.GetCustomAttributes(typeof(WindowsPhone), true).Length > 0)
                        {
                            //Add the property to list.
                            lstSerializableMembers.Add(memberInfo);
                        }
                        break;
                    //For web.
                    case "web":
                        //Check whether the attribute exist or not.
                        if (memberInfo.GetCustomAttributes(typeof(Web), true).Length > 0)
                        {
                            //Add the property to list.
                            lstSerializableMembers.Add(memberInfo);
                        }
                        break;
                    //Default.
                    default:
                        //Add the property to list.
                        lstSerializableMembers.Add(memberInfo);
                        break;
                }
            }
            //Return the list.
            return lstSerializableMembers;
        }
    }



Now we have to create the message formatter which is used  to send custom Message object to the client.For this we have to use  IDispatchMessageFormatter ‘s SerializeReply method.In which we can send custom Message object.

public class MessageFormatter : IDispatchMessageFormatter
    {
        //Formatter.
        private IDispatchMessageFormatter Formatter;

        //Costructor.
        public MessageFormatter(IDispatchMessageFormatter formatter)
        {
            //Set the Formatter here.
            this.Formatter = formatter;
        }

        public void DeserializeRequest(System.ServiceModel.Channels.Message message, object[] parameters)
        {
            //Do nothing.
        }

        public System.ServiceModel.Channels.Message SerializeReply(System.ServiceModel.Channels.MessageVersion messageVersion, object[] parameters, object result)
        {
            //Json body.
            byte[] body;

            //Json serializer.
            Newtonsoft.Json.JsonSerializer objJsonSerializer = new Newtonsoft.Json.JsonSerializer();

            //Set the ContractResolver which is used for getting serializable members based on User-Agent.
            objJsonSerializer.ContractResolver = new Resolver();

            //Memory stream.
            using (MemoryStream ms = new MemoryStream())
            {
                //Stream writer.
                using (StreamWriter sw = new StreamWriter(ms, Encoding.UTF8))
                {
                    //Json writer.
                    using (Newtonsoft.Json.JsonWriter objJsonWriter = new Newtonsoft.Json.JsonTextWriter(sw))
                    {
                        objJsonWriter.Formatting = Newtonsoft.Json.Formatting.Indented;

                        //Serialize the object to string.
                        objJsonSerializer.Serialize(objJsonWriter, result);

                        //Flush to writer.
                        sw.Flush();

                        //Get the json body.
                        body = ms.ToArray();
                    }
                }
            }

            //Create message object for client.
            Message replyMessage = Message.CreateMessage(messageVersion, "", new RawBodyWriter(body));

            //Add WebBodyFormatMessageProperty to message.
            replyMessage.Properties.Add(WebBodyFormatMessageProperty.Name, new WebBodyFormatMessageProperty(WebContentFormat.Raw));

            //HttpResponseMessageProperty.
            HttpResponseMessageProperty respProp = new HttpResponseMessageProperty();

            //Set the content type to application/json.
            respProp.Headers[HttpResponseHeader.ContentType] = "application/json";

            //Add HttpResponseMessageProperty to message.
            replyMessage.Properties.Add(HttpResponseMessageProperty.Name, respProp);

            //Return the message.
            return replyMessage;
        }
    }

public class RawBodyWriter : BodyWriter
    {
        byte[] content;
        public RawBodyWriter(byte[] content)
            : base(true)
        {
            this.content = content;
        }

        protected override void OnWriteBodyContents(XmlDictionaryWriter writer)
        {
            writer.WriteStartElement("Binary");
            writer.WriteBase64(content, 0, content.Length);
            writer.WriteEndElement();
        }
    }


We can add this MessageFormatter at different  levels.Here we are adding the MessageFormatter at endpoint level.For this we have to use IEndpointBehaviors’s ApplyDispatchBehavior method.

public class EndpointBehavior : IEndpointBehavior
    {
        public void AddBindingParameters(ServiceEndpoint endpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
        {
            //Do nothing.
        }

        public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
        {
            //Do nothing.
        }

        public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
        {
            //Iterate through all the operations in the dispatch runtime.
            foreach (DispatchOperation objDispatchOperation in endpointDispatcher.DispatchRuntime.Operations)
            {
                //Set the formatter here.
                objDispatchOperation.Formatter = new MessageFormatter(objDispatchOperation.Formatter);
            }
        }

        public void Validate(ServiceEndpoint endpoint)
        {
            //Do nothing.
        }
    }


Now host the service in console application.

//Create service host object.
ServiceHost objHost = new ServiceHost(typeof(Service), new Uri("http://localhost:1234/Service.svc/"));

            //Add endpoint with webHttpBinding.
            ServiceEndpoint ep = objHost.AddServiceEndpoint(typeof(IService), new WebHttpBinding(), "");

            //Add WebHttpBehavior to endpoint.
            ep.Behaviors.Add(new WebHttpBehavior() { HelpEnabled = true });

            //Add our custom behavior to endpoint.
            ep.Behaviors.Add(new EndpointBehavior());

            //Open service host.
            objHost.Open();

            Console.WriteLine("Service is running...");

            Console.ReadLine();

That’s it, by calling with different User-Agent’s we can get the different  propertys.

Let me know, if you have any feedback. Mail me for source code. Enjoy reading my articles…
sekhartechblog@gmail.com