How to can I serialize data to Communication between C++ zmq client and Python zmq Server

Question

UPDATING MY QUESTION

How to can I represent the arrived message in my python zmq server to show their content ?

According to this behavior, can I assume that the btnState data is sent to python server in anyway?

Context:

I am sending some data members structures using a C++ zeromq client process: ZMQComponent.h file

#include <zmq.hpp>
#include <sofa/defaulttype/VecTypes.h>

// To Quat datatype
#include <sofa/defaulttype/Quat.h>
using sofa::defaulttype::Quat;

using std::string;

namespace sofa
{

namespace component
{

namespace controller
{

/* data structure which I want send data to python zmq server */
struct instrumentData
{
  typedef sofa::defaulttype::Vec3d Vec3d;
  Vec3d pos;
  Quat quat;
  int btnState;
  float openInst;
  bool blnDataReady;
};

class ZMQComponent : public sofa::core::behavior::BaseController
{
  public:
    SOFA_CLASS(ZMQComponent, sofa::core::behavior::BaseController);

    ZMQComponent();
    virtual ~ZMQComponent();

    /* Conect to ZMQ external python Server  */
    void setupConnection();

    /* Send some data memeber instrumentData structure to ZMQ external Server  */
    void instrumentDataSend(instrumentData a);

    /* initialize function */
    void init();

};

} // namespace sofa

} // namespace component

} // namespace controller

The ZMQComponent.cpp is:

#include <sofa/core/ObjectFactory.h>
#include <zmq.hpp>
#include <iostream>
#include <string>
#include "ZMQComponent.h"


using namespace std;

namespace sofa
{

namespace component
{

namespace controller
{

/*  ZMQ Internal Client context and socket */
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REQ);

ZMQComponent::ZMQComponent(){}

void ZMQComponent::setupConnection()
{
    cout << "Connecting to python zeroMQ server ..." << endl;
    socket.connect("tcp://localhost:5555");
}

void ZMQComponent::instrumentDataSend(instrumentData a)
{
    /*  Initialize the data members structure instrumentData */
    a.pos = sofa::defaulttype::Vec3d(1.0f, 1.0f, 1.0f);
    a.quat = defaulttype::Quat(1.0f, 1.0f, 4.0f, 1.0f);
    a.btnState = 5671;
    a.openInst = 1.0f;
    a.blnDataReady = false;

    string s, test, result, d;
    s = to_string(a.btnState);
    test = " is a number";
    result = s + test;

    /*  We send  the btnState data  */
    zmq::message_t request(30);



/*  We ask for the memory address to ge the btnState content and send it. */
    memcpy(request.data(), &result, 30);
    socket.send(request);
}


/*  In the init function we create the objects to setup connection and send data  */
void ZMQComponent::init()
{
    std::cout << "ZeroMQCommunication::init()" << std::endl;
    ZMQComponent z;
    z.setupConnection();

    instrumentData itemp;
    z.instrumentDataSend(itemp);

}

/*  Other code related ....  */
ZMQComponent::~ZMQComponent(){}

// int ZeroMqComponentClass = sofa::core::RegisterObject("This component does nothing.").add<ZeroMqComponent>();
SOFA_DECL_CLASS(ZMQServerComponent)

int ZMQServerComponentClass = sofa::core::RegisterObject("This component create a Socket.").add< ZMQServerComponent >();
} // namespace controller

} // namespace component

} // namespace sofa

Then , my python zmq server which receive the btnState int variable is:

import time
import zmq

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
print('ZMQ Server listening ... ')

while True:
    #  Wait for next request from client
    message = socket.recv()
    print("Received message from Sofa: {}".format(message))

    #  Do some 'work'
    time.sleep(1)

The output or the message which arrive to python zmq server is the content of result variable (btnState turn to string in s content variable + string test concatenated) and some symbols characters of the :

(cnvss_test) ➜  Python git:(ZMQCommunication) ✗ python server.py
ZMQ Server listening ...
Received message from Sofa: b'\xb0\x1d\x19\xf4\xfd\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 is a number'

The previous output on my ZMQ python server script show that the string result from sofa is arrived to server, and their content is visualized, but too, this strings or characters symbols, which are product or consequence of the size of the zmq::message_t request(30) defined in my C++ client in the beginning.

If I assign a value less than 30 in the request, by example zmq::message_t request(10) the output in my server is:

Received message from Sofa: b'\x90\x94\xa1\x00\xfc\x7f\x00\x00\x0e\x00'

If I assign a value greater than 10 in the request, by example zmq::message_t request(20) the output in my server is:

Received message from Sofa: b'\x80$(\xc7\xfc\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 i

Then, the string or object which I receive in the server side, it has as long as the length or size assigned to zmq::message_t request variable

Based in the above mentioned, is ZMQ whom add this strings in my message received?

According to the previous process, my message is arrived to my server, then is correct attempt what the serialization process with some entity like protocol buffer is necessary? I understand that use something like google protocol buffer allow have some correlation more controlled in relation to objects sent and the objects received in relation of their real content ...

In any case, how to can I remove the strings or characters symbols that are added in the message arrived to the server?

Any support or orientation will be highly appreciated

If you put the question first ("How can I ... in the below ...?") and provide context as needed, then it is much easier to read and answer. — jpaugh
– jpaugh, Commented Jan 29, 2018 at 17:37

bazza · Accepted Answer · 2018-01-25 05:42:09Z

1

Your system is heterogeneous, meaning you need some sort of serialisation that is platform / language agnostic.

For your purposes the most convenient one to use is likely to be Google Protocol Buffers. This supports both C++ and Python very nicely. With this you'll be defining your messages / data structures in a schema file (file extension .proto), and using protoc to compile that to both C++ source code and also to Python source code. These give you classes that can serialise/deserialise to/from the same wireformat. The serialisation / deserialisation can nicely integrated with ZMQ message buffers.

There are others;

Apache Avro is a possibility.
I would avoid XSD schemas; in principle they're fine, but finding code generators that actually do a proper and complete job is difficult / expensive. For example, xsd.exe (from Microsoft) can compile an XSD schema to C++ classes (I think), but ignores the constraint fields in the schema (e.d. MinInclusive).
ASN1 is really good, but I've yet to find a decent implementation for Python. There is a code-first implementation of ASN.1 for python (pyasn), but that rather misses the whole point of having an ASN.1 schema...

answered Jan 25, 2018 at 5:42

bazza

8,61718 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

bgarcial Over a year ago

I'll try with Google Protocol Buffers. My sender is from C++ language and my reception is Python. What sort of installation recommend me you? Binaries or build protocol buffer along my C++ runtime? or o build protoc binary from source?

bazza Over a year ago

@bgarcial, I normally just down load the source and build protoc. Some Linux distros put out very old versions by default, so I prefer to not use them. Also it's worth integrating the use of protoc into your build process for C++, save you having to do it by hand every time. Good luck!

bgarcial Over a year ago

One more question. Why is necessary the serialization process? It is assumed that zeromq works via TCP and in the TCP protocol, the data does not undergo any transformation? This is related with the fact of that I have different platform/languages?

bazza Over a year ago

@bgarcial The serialisation is useful because you have both C++ and Python. ZMQ is simply a way of moving bytes from A to B. The way that C++ stores instrumentdata in memory is different to the way that Python would do it. Describe instrumentdata in a .proto schema, you end up with source code that gives you a class instrumentdata in both C++ and Python, and when serialised in C++ can be deserialised in Python to get the same content. WIthout that it is a lot of code to successfully interpret the byte stream from C++ into meaningful variables in Python (as you're beginning to find).

bazza Over a year ago

@bgarcial, think of GPB as being similar to Python's pickle that both C++ and Python (and Java and C# and lots of others) all understand.

|

Erwan Douaille · Accepted Answer · 2018-02-02 10:25:52Z

I'm using a similar zmq implementation in c++ with the sofa-framework (related to the autor plugin).

I don't have any issue sending data from zmq C++ to python zmq.

Here is an overview of my zmq server written in python :

import zmq

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.SUBSCRIBE, "")

print "Collecting data from c++ ..."
socket.connect ("tcp://127.0.0.1:6000")

while True:
    print socket.recv()

An here is an overview of the result :

douaille@douaille:~/Documents/zmq$ python zmqClient.py 
Collecting data from c++ ...
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1'

I'm using zmq as publisher, but it does not change anything for your needs. Plus the implementation for the request client is almost the same. It looks like you are printing the request and not the message

You can find an implementation of the c++ sender part here : https://github.com/SofaDefrost/sofa/blob/sofaCommunication/applications/plugins/Communication/components/serverCommunicationZMQ.inl

EDIT :

Here is an example using zmq request :

import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)

print("Collecting data from c++ using REQ ...")
socket.connect("tcp://localhost:6000")

while True:
    print("Sending request")
    socket.send(b"Hello")
    message = socket.recv()
    print("Received reply : %s" % message)

Result :

douaille@douaille:~/Documents/zmq$ python zmqClient.py 
Collecting data from c++ using REQ ...
Sending request
Received reply : /colorLight string:'1 1 1 1' 
Sending request
Received reply : /colorLight string:'1 1 1 1' 
Sending request
Received reply : /colorLight string:'1 1 1 1'

Collectives™ on Stack Overflow

How to can I serialize data to Communication between C++ zmq client and Python zmq Server

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related