fail to load searchd response / slice bounds out of range

Hi guys,

Hope you are all well !

I have 2 major issues with the go-sdk and managed to reproduce them into the following repository.
Ref. https://github.com/x0rzkov/manticore-gosdk-issue

In a nutshell, the two issues are:

1. On some queries, I have a slice bounds out of range error.
   To reproduce, please see the README.md of the repository above.

runtime error: slice bounds out of range [:774778400] with capacity 22412    
/usr/local/go/src/runtime/panic.go:106 (0x432eb2)    
	goPanicSliceAcap: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsSliceAcap})    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:311 (0x10a139c)    
	(*apibuf).getMysqlStrLen: result := string((*buf)[:lng])    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:234 (0x10a0a5a)    
	(*Sqlresult).parserow: strValue := buf.getMysqlStrLen()    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:74 (0x109f901)    
	(*Sqlresult).parseChain: if !rs.parserow(source) {    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:20 (0x10a62b5)    
	parseSphinxqlAnswer.func1: if rs.parseChain(answer) {    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/client.go:225 (0x1093649)    
	(*Client).netQuery: return parser(&answer), nil    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/manticore.go:398 (0x109503f)    
	(*Client).Sphinxql: blob, err := cl.netQuery(commandSphinxql,    
/home/ubuntu/lucmichalski/paper2code/main.go:3639 (0x118b2a7)    
	controllersSearch: res2, err2 := cl.Sphinxql(query)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/context.go:161 (0xc9b61a)    
	(*Context).Next: c.handlers[c.index](c)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/recovery.go:83 (0xcaedaf)    
	RecoveryWithWriter.func1: c.Next()    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/context.go:161 (0xc9b61a)    
	(*Context).Next: c.handlers[c.index](c)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/logger.go:241 (0xcadee0)    
	LoggerWithConfig.func1: c.Next()    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/context.go:161 (0xc9b61a)    
	(*Context).Next: c.handlers[c.index](c)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/gin.go:409 (0xca53f5)    
	(*Engine).handleHTTPRequest: c.Next()    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/gin.go:367 (0xca4b0c)    
	(*Engine).ServeHTTP: engine.handleHTTPRequest(c)    
/home/ubuntu/go/pkg/mod/github.com/qor/session@v0.0.0-20170907035918-8206b0adab70/gorilla/gorilla.go:125 (0xb2b591)    
	Gorilla.Middleware.func1: handler.ServeHTTP(w, req.WithContext(ctx))    
/usr/local/go/src/net/http/server.go:2012 (0x76eb73)    
	HandlerFunc.ServeHTTP: f(w, r)    
/usr/local/go/src/net/http/server.go:2807 (0x772002)    
	serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)    
/usr/local/go/src/net/http/server.go:1895 (0x76d97b)    
	(*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)    
/usr/local/go/src/runtime/asm_amd64.s:1373 (0x465670)    
	goexit: BYTE	``````x90	// NOP    
%CODE_BLOCK%`    
    
2. Sphinxql: failed to read searchd response    
When I dockerize manticore and make a query to manticore from the local env; everything works, and when I do exactly the same thing/query/task/action within dockerized application it triggers a failed to read searchd response.    
    
All the details to reproduce are also available in the README of the repository.    
    
Hope that you will find quickly the issues related to these bugs ^^    
    
Cheers,    
X   
   
 
 

[klirichek]

As I can guess by line numbers, that is code published some revisions ago.
In the near past it was fix about line length, and actual tip of master now is 7d072dc
Could you check, if your issue is still actual agains this head?

I just checked my code and the go.mod file, and it is on the right revision https://github.com/x0rzkov/manticore-gosdk-issue/blob/master/go.mod#L11

aka 7d072dc

I mean it is still actual

[klirichek]

ok, will check.
(I asked because according to your backtrace - line sphinxql.go:311 has no 'result := string((*buf)[:lng])' in 7d072dc, it was so in 0e8dbff. And that is quite important, since 7d072dc contains fix of old well-known crash in mysql processing, and that is why I asking about. (in 7d072dc that would be sphinxql.go:320)

Thanks, my whole website is blocked because of these 2 issues, so yeah if you can sort it out it gonna be awesome :-) ^^

Any updates mate ?

[klirichek]

Not yet, need to complete current backlog of C++ tasks first.

[githubmanticore]

fix error in mysql len parsing

That closes #15 via commit 70a49dc

Hi guys,

Thanks for the fix, I am testing it right now and will come back with a feedback during the day.

But remains the second bug, quiet annoying for my rss feeds based on manticoresearch:

When I dockerize manticore and make a query to manticore from the local env; everything works, 
and when I do exactly the same thing/query/task/action within dockerized application it 
triggers a failed to read searchd response.

I gave you all the details to reproduce here.
https://github.com/x0rzkov/manticore-gosdk-issue#second-bug-sphinxql-failed-to-read-searchd-response

Cheers,
X

[klirichek]

Sounds interesting to look, but right now I am not able to do it fast
(previous one I've reproduced on bare metal, running your repro in IDE on macbook where I have no docker engine. Will check later on linux with docker).
However just wonder: if you run queries as sphinxql commands - why don't you just use mysql proto directly?
"database/sql" works well to connect to daemon and run your queries directly, without need to proxy them via sphinxql api command.
And that is also looking as good way to check if the problem is really specific to api proto implementation (if connecting to the dockerized instance using database/sql on mysql port works, but api not), or to some internal network implementation details of the daemon (if connecting to real mysql works, but connect to manticore mysql port not), or some more generic missconfiguration (if neither works in the docker at all)

Hi,

I tried with the mysql driver it works but not for my search query as I am using multiple facets as arguments. I do not get facets information with the mysql driver.

Please check that issue as I have no clue how to debug that more than the repo I gave you.

Tell me that you found the issue as my whole website is stuck because of this last bug. ^^

Thanks in advance.

Cheers,
X

[sanikolaev]

@x0rzkov

I tried with the mysql driver it works but not for my search query as I am using multiple facets as arguments. I do not get facets information with the mysql driver.

https://github.com/go-sql-driver/mysql doesn't support multiple results, but https://stackoverflow.com/a/28577876/591867 works for me:

snikolaev@dev:~$ cat test_luc.go
package main

import (
    "flag"
    "fmt"

    "github.com/ziutek/mymysql/autorc"
    "github.com/ziutek/mymysql/mysql"
    _ "github.com/ziutek/mymysql/thrsafe"
)

type ScanFun func(int, []mysql.Row, mysql.Result) error

func RunSQL(hostport, user, pass, db, cmd string, scan ScanFun) error {
    conn := autorc.New("tcp", "", hostport, user, pass, db)

    err := conn.Reconnect()
    if err != nil {
        return err
    }

    res, err := conn.Raw.Start(cmd)
    if err != nil {
        return err
    }

    rows, err := res.GetRows()
    if err != nil {
        return err
    }

    RScount := 0
    scanErr := error(nil)

    for {
        if scanErr == nil {
            func() {
                defer func() {
                    if x := recover(); x != nil {
                        scanErr = fmt.Errorf("%v", x)
                    }
                }()
                scanErr = scan(RScount, rows, res)
            }()
        }

        if res.MoreResults() {
            res, err = res.NextResult()
            if err != nil {
                return err
            }
            rows, err = res.GetRows()
            if err != nil {
                return err
            }
        } else {
            break
        }

        RScount++
    }
    return scanErr
}

func main() {
    host := flag.String("host", "localhost:9306", "define the host where the db is")
    user := flag.String("user", "root", "define the user to connect as")
    pass := flag.String("pass", "", "define the pass to use")
    db := flag.String("db", "information_schema", "what db to default to")

    sql := flag.String("sql", "select * from t where match('abc') facet a", "Query to run")

    flag.Parse()

    scan := func(rcount int, rows []mysql.Row, res mysql.Result) error {
        if res.StatusOnly() {
            return nil
        }

        for idx, row := range rows {
            fmt.Print(rcount, "-", idx, ") ")
            for i, _ := range row {
                fmt.Print(row.Str(i))
                fmt.Print(" ")
            }
            fmt.Println("")
        }
        return nil
    }

    fmt.Println("Host - ", *host)
    fmt.Println("Db   - ", *db)
    fmt.Println("User - ", *user)

    if err := RunSQL(*host, *user, *pass, *db, *sql, scan); err != nil {
        fmt.Println(err)
    }
}
snikolaev@dev:~$ go get github.com/ziutek/mymysql/autorc
snikolaev@dev:~$ go run test_luc.go
Host -  localhost:9306
Db   -  information_schema
User -  root
0-0) 2810903486202382289 0 1
0-1) 2810903486202382290 1 0
0-2) 2810903486202382291 1 2
1-0) 0 1
1-1) 1 2

Hi guys,

Updated my code for using your snippets, and it works from the docker container:
https://paper2code.com/feed?f_subjects=5,4

Sounds like docker is not the cause. Isn't it ?

Cheers,
X