eLua net module/UIP & multiple connections

[Michael Sallaway]

Hi all,

I've developed a board based on an LM3S6950, and added support for it to
eLua -- it's basically acting as a giant ethernet->RS485 converter, with
different streams for each of the 3 different UARTs.

I've got most of it working -- it's running eLua on the platform, I've
got a console, and it's run the lhttpd test server fine, so I know all
the base components are working properly. However, when I run my script,
it's not accepting connections most of the time, and when it does, it
doesn't process any data.

I've copied the relevant parts of my script below. I'm basically using
the timeout specifiers to the net.accept() and net.recv() calls to poll
for new connections, and then poll for data on each open socket.
However, I'm finding that when I initiate a connection from another
machine (using netcat to dump a fixed packet over the network to the dev
board), it's only accepted about 20% of the time (guesstimate), and when
it is accepted, it doesn't ever process any of the data.

So, my question(s) are thus: for the networking implementation, does the
lua script have to be waiting/blocking in the net.accept() or net.recv()
call to actually receive or process any data? Or does the underlying
network system queue data until it's requested?  If so, then does anyone
have any suggestions as to what may be causing my problems?

Thanks for any advice or comments anyone can give!

Cheers,
Michael




while true do
     -- check to see if there is anyone trying to connect (port 7474,
and use timer 2 to timeout after 10ms)
     local sock, rip, acceptErr = net.accept(7474, 2, 10000)

     -- if we got a connection, remember it
     if (acceptErr == net.ERR_OK) then
         local newEntry = { socket = sock, remoteip = rip }
         table.insert(sockets, newEntry)
         print("Got connection on socket: ", sock)
         print("Remote ip: " .. net.unpackip(rip, "*s"))
     end

     -- iterate through all sockets, see if there's any data, and if so,
update the states as appropriate
     for i,v in ipairs(sockets) do
         local socket = v["socket"]
         local remoteip = v["remoteip"]
         local data = { }

         -- listen for 10ms (on timer 2) for up to 2048 bytes from this
socket
         data, readErr = net.recv(socket, 2048, 2, 10000)
         if (readErr == net.ERR_OK) then
             print("received data, processing # bytes: ", #data)
             processPacket(data);
         elseif (readErr == net.ERR_CLOSED) then
             -- close and forget the socket
             print("socket closed, deleting: ", socket)
             net.close(socket)
             table.remove(sockets, i)

         elseif (readErr == net.ERR_ABORTED) then
             print("error: socket read aborted:", socket)

         elseif (readErr == net.ERR_OVERFLOW) then
             print("error: socket read overflowed: ", socket)

         elseif (readErr == net.ERR_TIMEDOUT or readErr ==
net.ERR_TIMEOUT) then
             -- typo in the API; it can be either (TIMED or TIME)OUT.
         else
             print("error: don't know what the read result code was!
socket / resultCode: ", socket, readErr)
         end
     end

     for k,v in pairs(outputPackets) do
         -- if the outputPackets table for this stream isn't empty, and
the UART Tx is empty, start a packet
         if (next(v) ~= nil) then
             if (lm3s.uart.txEmpty(k - 1)) then
                 print("Empty Tx for UART, will start new packet: ", k-1);
                 uartTxInterrupt(k - 1);
             end
         end
     end

     collectgarbage('collect')
end

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hi,

On Thu, Sep 22, 2011 at 11:44 AM, Michael Sallaway <[hidden email]> wrote:

Hi all,

I've developed a board based on an LM3S6950, and added support for it to eLua -- it's basically acting as a giant ethernet->RS485 converter, with different streams for each of the 3 different UARTs.

I've got most of it working -- it's running eLua on the platform, I've got a console, and it's run the lhttpd test server fine, so I know all the base components are working properly. However, when I run my script, it's not accepting connections most of the time, and when it does, it doesn't process any data.

I've copied the relevant parts of my script below. I'm basically using the timeout specifiers to the net.accept() and net.recv() calls to poll for new connections, and then poll for data on each open socket. However, I'm finding that when I initiate a connection from another machine (using netcat to dump a fixed packet over the network to the dev board), it's only accepted about 20% of the time (guesstimate), and when it is accepted, it doesn't ever process any of the data.

So, my question(s) are thus: for the networking implementation, does the lua script have to be waiting/blocking in the net.accept() or net.recv() call to actually receive or process any data? Or does the underlying network system queue data until it's requested?  If so, then does anyone have any suggestions as to what may be causing my problems?

The short answer to your question is "the TCP/IP implementation in eLua is buggy". So buggy, in fact, that it sometimes can simply loose packages. I've just discovered this recently. I suspect this is the case with your application. I have a fix in the works, but I can't estimate when it will be ready.

Best,

Bogdan

 

while true do
   -- check to see if there is anyone trying to connect (port 7474, and use timer 2 to timeout after 10ms)
   local sock, rip, acceptErr = net.accept(7474, 2, 10000)

   -- if we got a connection, remember it
   if (acceptErr == net.ERR_OK) then
       local newEntry = { socket = sock, remoteip = rip }
       table.insert(sockets, newEntry)
       print("Got connection on socket: ", sock)
       print("Remote ip: " .. net.unpackip(rip, "*s"))
   end

   -- iterate through all sockets, see if there's any data, and if so, update the states as appropriate
   for i,v in ipairs(sockets) do
       local socket = v["socket"]
       local remoteip = v["remoteip"]
       local data = { }

       -- listen for 10ms (on timer 2) for up to 2048 bytes from this socket
       data, readErr = net.recv(socket, 2048, 2, 10000)
       if (readErr == net.ERR_OK) then
           print("received data, processing # bytes: ", #data)
           processPacket(data);
       elseif (readErr == net.ERR_CLOSED) then
           -- close and forget the socket
           print("socket closed, deleting: ", socket)
           net.close(socket)
           table.remove(sockets, i)

       elseif (readErr == net.ERR_ABORTED) then
           print("error: socket read aborted:", socket)

       elseif (readErr == net.ERR_OVERFLOW) then
           print("error: socket read overflowed: ", socket)

       elseif (readErr == net.ERR_TIMEDOUT or readErr == net.ERR_TIMEOUT) then
           -- typo in the API; it can be either (TIMED or TIME)OUT.
       else
           print("error: don't know what the read result code was! socket / resultCode: ", socket, readErr)
       end
   end

   for k,v in pairs(outputPackets) do
       -- if the outputPackets table for this stream isn't empty, and the UART Tx is empty, start a packet
       if (next(v) ~= nil) then
           if (lm3s.uart.txEmpty(k - 1)) then
               print("Empty Tx for UART, will start new packet: ", k-1);
               uartTxInterrupt(k - 1);
           end
       end
   end

   collectgarbage('collect')
end

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Michael Sallaway]

On 27/09/11 17:15, Bogdan Marinescu wrote:
>
> The short answer to your question is "the TCP/IP implementation in
> eLua is buggy". So buggy, in fact, that it sometimes can simply loose
> packages. I've just discovered this recently. I suspect this is the
> case with your application.

Aah. Well, that'll explain it, I guess. :-)  Damn.

Do you have any more information about it? I'm more than happy to crack
it open and start playing around to find a fix (I spent a good few days
getting familiar with the underlying bits and pieces to write my stuff,
so I'm no expert, but I'm a bit familiar with it), but would have no
idea where to start.


> I have a fix in the works, but I can't estimate when it will be ready.

Is there any way I can help with it at all? Or, are there any
alternatives available?

Or even any workarounds if you know where the problem lies. I'm happy to
block on the accept() call if needed (as the sample httpd script works
fine, every time), if that would avoid the problem.

Thanks,
Michael


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Tue, Sep 27, 2011 at 10:42 AM, Michael Sallaway <[hidden email]> wrote:

On 27/09/11 17:15, Bogdan Marinescu wrote:

The short answer to your question is "the TCP/IP implementation in eLua is buggy". So buggy, in fact, that it sometimes can simply loose packages. I've just discovered this recently. I suspect this is the case with your application.

Aah. Well, that'll explain it, I guess. :-)  Damn.

Do you have any more information about it? I'm more than happy to crack it open and start playing around to find a fix (I spent a good few days getting familiar with the underlying bits and pieces to write my stuff, so I'm no expert, but I'm a bit familiar with it), but would have no idea where to start.

There are more issues:

- some coding problems in the eLua uIP "adapter" (src/elua_uip.c)  prevent data from being properly received because the uIP interaction isn't properly handled. These errors (or at least most of them) were fixed.

- uIP itself is a "fire and forget" stack by design: it gives you data and it expects you do something with it immediately. If more data arrives (on the same socket or on a different socket) the old data is overwritten. The solution to this was to add per-socket buffers. This means increased memory consumption, of course, but it also means that it's much harder to loose data.

That said, the "accept" part still doesn't work very well. It's OK most of the time, but sometimes it doesn't respond to a request or responds very slowly, and I've yet to understand why. I'm starting to think that trying so hard to turn uIP into something it's not is just not the right thing to do here. Ideally I'd be looking into lwIP, but I really don't have enough time for that.

With all this in mind, I'll try to integrate my TCP/IP changes to eLua (into a separate branch for now), maybe this helps in your particular case. But again, I simply can't estimate when I'll be able to do this.

Best,

Bogdan

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Michael Sallaway]

On 27/09/11 17:57, Bogdan Marinescu wrote:

> - some coding problems in the eLua uIP "adapter" (src/elua_uip.c)
>  prevent data from being properly received because the uIP interaction
> isn't properly handled. These errors (or at least most of them) were
> fixed.
> - uIP itself is a "fire and forget" stack by design: it gives you data
> and it expects you do something with it immediately. If more data
> arrives (on the same socket or on a different socket) the old data is
> overwritten. The solution to this was to add per-socket buffers. This
> means increased memory consumption, of course, but it also means that
> it's much harder to loose data.

So does this mean it's more likely to be happy and work properly if the
lua script blocks on the net.accept()? (and only uses a single
socket/connection at a time)?

I'm happy to run with that if it's the best workaround for now, just
trying to get a feel for how it works.



> That said, the "accept" part still doesn't work very well. It's OK
> most of the time, but sometimes it doesn't respond to a request or
> responds very slowly, and I've yet to understand why. I'm starting to
> think that trying so hard to turn uIP into something it's not is just
> not the right thing to do here. Ideally I'd be looking into lwIP, but
> I really don't have enough time for that.
> With all this in mind, I'll try to integrate my TCP/IP changes to eLua
> (into a separate branch for now), maybe this helps in your particular
> case. But again, I simply can't estimate when I'll be able to do this.

No worries, that would be fantastic if you're able to. Greatly appreciated.

Cheers,
Michael

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Tue, Sep 27, 2011 at 11:05 AM, Michael Sallaway <[hidden email]> wrote:

On 27/09/11 17:57, Bogdan Marinescu wrote:

- some coding problems in the eLua uIP "adapter" (src/elua_uip.c)  prevent data from being properly received because the uIP interaction isn't properly handled. These errors (or at least most of them) were fixed.
- uIP itself is a "fire and forget" stack by design: it gives you data and it expects you do something with it immediately. If more data arrives (on the same socket or on a different socket) the old data is overwritten. The solution to this was to add per-socket buffers. This means increased memory consumption, of course, but it also means that it's much harder to loose data.

So does this mean it's more likely to be happy and work properly if the lua script blocks on the net.accept()? (and only uses a single socket/connection at a time)?\

Yes. It also means that you might loose data even in this scenario, however it's less likely. 

 

I'm happy to run with that if it's the best workaround for now, just trying to get a feel for how it works.

Please try with blocking accept and let me know how it works for you.

Best,

Bogdan

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev