September 2018 – Records And Memorys

September 28, 2018September 28, 2018

ZMQ 各种 socket 介绍

- router – dealer 模式

为什么有 router – dealer 模式 ? 处理多client , 多service 的情况

例如下图：

1个client,多个service , 最简单的设计方法是： connect each client socket to multiple service endpoints

在这种情况下：

One client socket can connect to multiple service sockets, and the REQ socket will then distribute requests among these services. Let’s say you connect a client socket to three service endpoints; A, B, and C. The client makes requests R1, R2, R3, R4. R1 and R4 go to service A, R2 goes to B, and R3 goes to service C.

This design lets you add more clients cheaply. You can also add more services. Each client will distribute its requests to the services. But each client has to know the service topology. If you have 100 clients and then you decide to add three more services, you need to reconfigure and restart 100 clients in order for the clients to know about the three new services.

不好的地方是： client必须知道所有的service ，当新增若干个service，每一个client 都要修改并重启

router 或者 dealer 就是在client 和 service 之间的中间层 , 用 zmq_poll() 实现： What we want is something sitting in between clients and services that centralizes all knowledge of the topology. Ideally, we should be able to add and remove services or clients at any time without touching any other part of the topology.

实现示例图是这样的：

In this simple extended request-reply pattern, REQ talks to ROUTER and DEALER talks to REP.

来源文档：

http://zguide.zeromq.org/page:all

September 28, 2018

Maven Note

pom reference : https://www.apache.org/

September 28, 2018

How to use ZMQ

simple example :

server side : http://zguide.zeromq.org/java:hwserver

client side : http://zguide.zeromq.org/java:hwclient

server side code

package useZMQ;

import org.apache.log4j.Logger;
import org.zeromq.ZMQ;
//import org.zeromq.ZMQ.Context;
//import org.zeromq.ZMQ.Socket;

public class ZMQServer {
	static Logger logger = Logger.getLogger(ZMQServer.class);
	
    public static void main(String[] args) throws Exception {
    	
        ZMQ.Context context = ZMQ.context(1);

        //  Socket to talk to clients
        ZMQ.Socket responder = context.socket(ZMQ.REP);
        responder.bind("tcp://*:5555");

        while (!Thread.currentThread().isInterrupted()) {
            // Wait for next request from the client
            byte[] request = responder.recv(0);
            //System.out.println("Received " + new String (request));
            
            logger.info("Received " + new String (request));
            // Do some 'work'
            Thread.sleep(1000);

            // Send reply back to client
            String reply = "World";
            responder.send(reply.getBytes(), 0);
        }
        responder.close();
        context.term();
    }

}

package useZMQ;

import org.apache.log4j.Logger;

import org.zeromq.ZMQ;

//import org.zeromq.ZMQ.Context;

//import org.zeromq.ZMQ.Socket;

public class ZMQServer {

static Logger logger = Logger.getLogger(ZMQServer.class);

public static void main(String[] args) throws Exception {

ZMQ.Context context = ZMQ.context(1);

// Socket to talk to clients

ZMQ.Socket responder = context.socket(ZMQ.REP);

responder.bind("tcp://*:5555");

while (!Thread.currentThread().isInterrupted()) {

// Wait for next request from the client

byte[] request = responder.recv(0);

//System.out.println("Received " + new String (request));

logger.info("Received " + new String (request));

// Do some 'work'

Thread.sleep(1000);

// Send reply back to client

String reply = "World";

responder.send(reply.getBytes(), 0);

}

responder.close();

context.term();

}

client code :

package useZMQClient;

import org.zeromq.ZMQ;

public class ZMQClient {
	
    public static void main(String[] args) {
        ZMQ.Context context = ZMQ.context(1);

        //  Socket to talk to server
        System.out.println("Connecting to hello world server…");

        ZMQ.Socket requester = context.socket(ZMQ.REQ);
        requester.connect("tcp://176.122.178.37:5555");

        for (int requestNbr = 0; requestNbr != 10; requestNbr++) {
            String request = "Hello";
            System.out.println("Sending Hello " + requestNbr);
            requester.send(request.getBytes(), 0);

            byte[] reply = requester.recv(0);
            System.out.println("Received " + new String(reply) + " " + requestNbr);
        }
        requester.close();
        context.term();
    }
    

}

package useZMQClient;

import org.zeromq.ZMQ;

public class ZMQClient {

public static void main(String[] args) {

ZMQ.Context context = ZMQ.context(1);

// Socket to talk to server

System.out.println("Connecting to hello world server…");

ZMQ.Socket requester = context.socket(ZMQ.REQ);

requester.connect("tcp://176.122.178.37:5555");

for (int requestNbr = 0; requestNbr != 10; requestNbr++) {

String request = "Hello";

System.out.println("Sending Hello " + requestNbr);

requester.send(request.getBytes(), 0);

byte[] reply = requester.recv(0);

System.out.println("Received " + new String(reply) + " " + requestNbr);

}

requester.close();

context.term();

}

server pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>Ly.java.program</groupId>
  <artifactId>useZMQ</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  
  <dependencies>
		<dependency>
			<groupId>org.zeromq</groupId>
			<artifactId>jeromq</artifactId>
			<version>0.3.4</version>
		</dependency>

		<dependency>
			<groupId>log4j</groupId>
			<artifactId>log4j</artifactId>
			<version>1.2.17</version>
		</dependency>
    </dependencies>
  
  	
  	<build>
  		<resources>
			<resource>
				<directory>src/main/resources</directory>
				<targetPath>${project.build.directory}</targetPath>
			</resource>
		</resources>
 
  		<defaultGoal>install</defaultGoal>
  		
  		<plugins>

			<!-- Set a JDK compiler level -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.0</version>
				<configuration>
					<source>1.7</source>
					<target>1.7</target>
				</configuration>
			</plugin>
			
			<!-- Make this jar executable -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-jar-plugin</artifactId>
				<configuration>
					<archive>
						<manifest>
							<addClasspath>true</addClasspath>
							<classpathPrefix>lib/</classpathPrefix>
							<mainClass>useZMQ.ZMQServer</mainClass>
						</manifest>
						<manifestEntries>
							<Class-Path>.</Class-Path>
						</manifestEntries>
					</archive>
				</configuration>
			</plugin>
			
		</plugins>
		
  		
  	</build>
  
</project>

<groupId>Ly.java.program</groupId>

<artifactId>useZMQ</artifactId>

<version>0.0.1-SNAPSHOT</version>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

</properties>

<groupId>org.zeromq</groupId>

<artifactId>jeromq</artifactId>

</dependency>

</dependency>

</dependencies>

<build>

<directory>src/main/resources</directory>

<targetPath>${project.build.directory}</targetPath>

</resource>

</resources>

<defaultGoal>install</defaultGoal>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-compiler-plugin</artifactId>

</configuration>

</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-jar-plugin</artifactId>

<mainClass>useZMQ.ZMQServer</mainClass>

</manifest>

<Class-Path>.</Class-Path>

</manifestEntries>

</archive>

</configuration>

</plugin>

</plugins>

</build>

</project>

September 26, 2018

Eclipse with spring framwork , use gradle to compile , How to use log4j

For waqas ‘s diameter-relay project , It can output logs to /logs/out.log .

the project is a Spring boot project , use gradle to compile , then the problem is : How to enable Log4j on SpringBoot Gradle: log4j-over-slf4j.jar, slf4j-log4j12.jar Multiple Bindings

But, In diameter-relay project , I neither find any info in build.gradle , nor in gradlew

September 26, 2018

How to search a certain word through whole project in eclipse

same problem from stackoverflow is :

How to search in Eclipse through project for some word ? When I start CTRL-h I need to choose field, method or …., but I need to find all occurences in project no metter method, field or .. ! How to achieve that ?

Answer is :

CTRL + H -> File Search -> Containing Text ‘some word’
make sure the scope is set to workspace or enclosing projects. Choosing Selected Resources will narrow the result set to the folder or file you selected in the explorer.

Further if you know the file type that might contain the text you can use the File Name patterns with values like *.php, *.phtml, *.html

September 25, 2018September 26, 2018

word

landslide — 一方选票占压倒多数的选举；山体滑坡

reciprocal — 互惠的

spectecular — 壮观精彩

September 10, 2018September 10, 2018

How to check thrift server status

If one function in server need 10 seconds to finish , and client one time send 100 request , How to know how many request is pending ?

September 9, 2018September 11, 2018

Thrift introduce

data type

bool A boolean value, true or false
byte A signed byte
i16 A 16-bit signed integer
i32 A 32-bit signed integer
i64 A 64-bit signed integer
double A 64-bit floating point number
string An encoding-agnostic text or binary string

struct

A struct is essentially equivalent to a class in object oriented
programming languages. A struct has a set of strongly typed
fields, each with a unique name identifier. The basic syntax for
defining a Thrift struct looks very similar to a C struct definition.
Fields may be annotated with an integer field identifier (unique to
the scope of that struct) and optional default values. Field identifiers
will be automatically assigned if omitted, though they are strongly
encouraged for versioning reasons discussed later.

Example of struct

    struct Work {
             1: i32 num1 = 0,
             2: i32 num2,
             3: Operation op,
             4: optional string comment,
      }

struct Work {

1: i32 num1 = 0,

2: i32 num2,

3: Operation op,

4: optional string comment,

}

struct Example {
           1:i32 number=10,
           2:i64 bigNumber,

           3:double decimals,
           4:string name="thrifty"
    }

struct Example {

1:i32 number=10,

2:i64 bigNumber,

3:double decimals,

4:string name="thrifty"

}

Service

Services are defined using Thrift types. Definition of a service is
semantically equivalent to defining an interface (or a pure virtual
abstract class) in object oriented programming. The Thrift compiler
generates fully functional client and server stubs that implement the
interface. Services are defined as follows:

service <name> {
<returntype> <name>(<arguments>)
[throws (<exceptions>)]
...
}

service <name> {

<returntype> <name>(<arguments>)

[throws (<exceptions>)]

...

}

An example is :

service StringCache {
void set(1:i32 key, 2:string value),
string get(1:i32 key) throws (1:KeyNotFound knf),
void delete(1:i32 key)
}

service StringCache {

void set(1:i32 key, 2:string value),

string get(1:i32 key) throws (1:KeyNotFound knf),

void delete(1:i32 key)

}

Note : an async modifier keyword may be added to a void function, which will generate code that does not wait for a response from the server.

Servers and Multithreading

Thrift services require basic multithreading to handle simultaneous
requests from multiple clients

**** Important

Now I will create a RouteServer which supply a “QueryRoute” function , client send Anumber and Bnumber as parameter , server will reply a struct include 2 int and 1 string , how to write this service ?

First define the struct in thrift file

struct Ret {
  1: i32 num1 = 0,
  2: i32 num2,
  3: string routeID,
}

struct Ret {

1: i32 num1 = 0,

2: i32 num2,

3: string routeID,

}

then define the interface

service RouteServer {
Ret QueryRoute(1:string Anumber, 2:string Bnumber)
}

service RouteServer {

Ret QueryRoute(1:string Anumber, 2:string Bnumber)

}

Full thrift file is : name is “RouteServer.thrift”

namespace netcore tutorial    # only gen cSharp server and client 

exception InvalidOperation {
  1: i32 whatOp,
  2: string why
}

struct Ret {
  1: i32 num1 = 0,
  2: i32 num2,
  3: string routeID,
}


service RouteServer {
Ret QueryRoute(1:string Anumber, 2:string Bnumber)
}

namespace netcore tutorial # only gen cSharp server and client

exception InvalidOperation {

1: i32 whatOp,

2: string why

}

struct Ret {

1: i32 num1 = 0,

2: i32 num2,

3: string routeID,

}

service RouteServer {

Ret QueryRoute(1:string Anumber, 2:string Bnumber)

}

goto folder : /home/liuyang/thrift-0.11.0/tutorial

run command : rm -r gen-csharp , delete existing folder

then run command : thrift -r –gen csharp RouteServer.thrift

Then goto /home/liuyang/thrift-0.11.0/tutorial/gen-csharp , copy “RouteServer.cs” and “InvalidOperation.cs” and “Ret.cs” to windows machine console application , add to project .

For server side “Program.cs” ,change it to

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Thrift.Server;
using Thrift.Transport;


namespace RouteService
{
    public class RouteServerHandler : RouteServer.Iface
    {
        Dictionary<int, SharedStruct> log;

        public RouteServerHandler()
        {
            log = new Dictionary<int, SharedStruct>();
        }

        public SharedStruct getStruct(int key)
        {
            //Console.WriteLine("getStruct({0})", key);
            return log[key];
        }

        public Ret QueryRoute(string anumber, string bnumber)
        {
            Ret r1 = new Ret();
            r1.Num1 = 1;
            r1.Num2 = 2;
            r1.RouteID = "MaxisRoute";

            return r1;
        }
    }


    class Program
    {

        static void Main(string[] args)
        {
            try
            {
                RouteServerHandler handler = new RouteServerHandler();
                RouteServer.Processor processor = new RouteServer.Processor(handler);
                TServerTransport serverTransport = new TServerSocket(9090);


                //TServer server = new TSimpleServer(processor, serverTransport);
                TServer server = new TThreadPoolServer(processor, serverTransport);




                // Use this for a multithreaded server
                // server = new TThreadPoolServer(processor, serverTransport);

                Console.WriteLine("Starting the server...");
                server.Serve();
            }
            catch (Exception x)
            {
                Console.WriteLine(x.StackTrace);
            }
            Console.WriteLine("done.");
            Console.ReadLine();

        }
    }
}

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

using Thrift.Server;

using Thrift.Transport;

namespace RouteService

{

public class RouteServerHandler : RouteServer.Iface

{

Dictionary<int, SharedStruct> log;

public RouteServerHandler()

{

log = new Dictionary<int, SharedStruct>();

}

public SharedStruct getStruct(int key)

{

//Console.WriteLine("getStruct({0})", key);

return log[key];

}

public Ret QueryRoute(string anumber, string bnumber)

{

Ret r1 = new Ret();

r1.Num1 = 1;

r1.Num2 = 2;

r1.RouteID = "MaxisRoute";

return r1;

}

class Program

{

static void Main(string[] args)

{

try

{

RouteServerHandler handler = new RouteServerHandler();

RouteServer.Processor processor = new RouteServer.Processor(handler);

TServerTransport serverTransport = new TServerSocket(9090);

//TServer server = new TSimpleServer(processor, serverTransport);

TServer server = new TThreadPoolServer(processor, serverTransport);

// Use this for a multithreaded server

// server = new TThreadPoolServer(processor, serverTransport);

Console.WriteLine("Starting the server...");

server.Serve();

}

catch (Exception x)

{

Console.WriteLine(x.StackTrace);

}

Console.WriteLine("done.");

Console.ReadLine();

}

For client “Program.cs” ,change it to

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Thrift;
using Thrift.Protocol;
using Thrift.Server;
using Thrift.Transport;


namespace RouteClient
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                TTransport transport = new TSocket("localhost", 9090);
                TProtocol protocol = new TBinaryProtocol(transport);
                RouteServer.Client client = new RouteServer.Client(protocol);

                transport.Open();
                try
                {

                    string s1 = "";
                    string s2 = "";

                    Ret r2 = client.QueryRoute(s1, s2);

                    s1 = "";

                }
                finally
                {
                    transport.Close();
                }
            }
            catch (TApplicationException x)
            {
                Console.WriteLine(x.StackTrace);
            }

        }
    }
}

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

using Thrift;

using Thrift.Protocol;

using Thrift.Server;

using Thrift.Transport;

namespace RouteClient

{

class Program

{

static void Main(string[] args)

{

try

{

TTransport transport = new TSocket("localhost", 9090);

TProtocol protocol = new TBinaryProtocol(transport);

RouteServer.Client client = new RouteServer.Client(protocol);

transport.Open();

try

{

string s1 = "";

string s2 = "";

Ret r2 = client.QueryRoute(s1, s2);

s1 = "";

}

finally

{

transport.Close();

}

catch (TApplicationException x)

{

Console.WriteLine(x.StackTrace);

}

Now client can call server function and get reply .

September 7, 2018September 7, 2018

How to use thrift call sql server procedure and return result to client

How server side call sql server SP

How client send parameter to server

How server reply to client

Ref article

Setting up a Thrift Server

https://medium.com/@saipeddy/setting-up-a-thrift-server-4eb0c55c11f0