Reading arbitrary number of bytes? (Whois client)

[njh]

Hi,

I am looking into how to implement a Whois client using Net::TCPClient.
However I am a bit stuck, because I can't work out how to read an arbitrary number of bytes.

The Whois protocol involves:

1. Opening a TCP socket
2. Sending the request string
3. Reading all the bytes returned until the socket is closed

I guess it is quite similar to very basic HTTP.

If I try reading a large number of bytes, like this:

Net::TCPClient.connect(server: 'whois.nic.me:43') do |client|
  client.write("DOMAIN njh.me\r\n")
  response = client.read(4096)
  puts "Received: #{response}"
end

Then I get an error:

/Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/net_tcp_client-2.2.0/lib/net/tcp_client/tcp_client.rb:651:in `socket_read': Connection lost while reading data (Net::TCPClient::ConnectionFailure)

Because the server closed the connection before the client received the full 4096 bytes. But if I set the read size, to a low number (for example 100), then I only get back those 100 bytes.

I have considered having a loop reading a small number of bytes at a time, but I can't see how to avoid the Net::TCPClient::ConnectionFailure error, given that I don't know how many bytes the server is going to send.

The reason for using Net::TCPClient at all and not just a plain TCPSocket, is because I want to try each of the IP addresses in DNS in turn and not just fail if the first chosen IP address is down.

Any help would be very appreciated.

[reidmorrison]

Good point, does not look like the API takes into consideration the use case where we just want call socket.read and just read everything until the socket is closed.
Can you check if using a regular socket works in the above scenario when calling socket.read?
If so, I will update the gem to support a read with no parameters.

[njh]

Yes, you can call IO#read without any parameters:

If length is omitted or is nil, it reads until EOF and the encoding conversion is applied, if applicable. A string is returned even if EOF is encountered before any data is read.

Here is an example minimal Whois client:

TCPSocket.open("whois.nic.me", 43) do |sock|
  sock.write("DOMAIN njh.me\r\n")
  response = sock.read
  puts "Received: #{response}"
end

It would also be really useful to be able to read line by line (IO#gets / IO#each_line / IO#readlines).

Here is an example minimal HTTP client:

TCPSocket.open("www.bbc.co.uk", 80) do |sock|
  sock.write("GET / HTTP/1.0\nHost: www.bbc.co.uk\n\r\n")
  sock.each_line do |line|
    line.chomp!
    break if line.empty?
    puts "Header: #{line}"
  end
  puts "Body: #{sock.read}"
end

It reads headers line by line, until a blank line and then reads the body until EOF.

[reidmorrison]

Want to submit a pull request adding both capabilities?

[njh]

I have taken a quick look and it doesn't look straightforward. While the length parameter for IO#read is optional, the length parameter for IO#read_nonblock is not. I am not sure I understand what you are doing in your code well enough to work out how to add support for this.

I was having trouble with IPv6 being down for some public whois server - so I was looking at Net::TCPClient for use in the whois ruby gem. Hower the author is not keen on adding an external dependency. The problem is probably solved just by iterating over the addresses using Resolv.each_address.