XHTTP: rename X-Padding to X-Signature, add xPaddingQueryParam

[paqx]

Hi!

As you might know, there are currently white lists for mobile internet in many regions of Russia. It's hard set up an xray server to bypass the restriction because you need either a white-listed IP or a white-listed CDN that supports POST requests. This commit targets the latter case. There are few white listed CDNs in Russia and unfortunately they try to block xray requests. I myself bumped into one such case. CDNVideo started to block any POST requests whose query string matches the pattern x_padding=XXXXX. Also, some requests included an X-Padding header. Both the query param and header have a very unusual name and can be easily spotted. My goal was to bypass this by allowing users to change the name of the query param. I ran a test and successfully bypassed CDNVideo's filter. It's not a perfect solution because they can start blocking requests by the value but anyway.

What I did:

1. Added xPaddingQueryParam. It can be set in the config to change the name of the query param used to pass the padding data. The default is x_padding
2. Renamed X-Padding header to X-Signature - a lot more common and widely used by actual CDN users.
3. Changed the way the padding data is generated. It now consists of chars X and Z, not only X.

Further improvements:

• Other CDNs disabled POST requests but PUT and PATCH requests are often not blocked. I plan to allow users to choose an HTTP method to pass data.
• The padding value is still very noticable. I don't understand the idea well enough yet but it the post said X was chosen because it's compressed into 8 bits. I checked the encoding table and the only suitable alternative was char Z. Not much and I wonder if it's possible to make the data look more random. In this case it will be very hard to block it because it can disguise as cache busters.
• Use a more realistic user agent. I noticed that the requests are currently passed with the Go-http-client/2.0 UA. Also doesn't look like a real CDN user.

Thanks

[Fangliding]

这个部分暂时不需要pr 不过它是计划中的 #4346 (comment)

不用X换用随机ASCII字符的方法非常简单 只需要确定字符集(eg base62)然后计算它在哈夫曼编码下的压缩率然后将预期的xpadding长度除以它做修正就行了 最后的结果会趋近xpadding的长度而且还会额外附加一点随机性 这早在全部用X填充之前我就想到了 不过问题不大而且这个字母占8个位确实很有缘所以当初就没提这一点(当然复杂一点的话精确长度也是完全可以做到的)

[hxehex]

这个部分暂时不需要pr 不过它是计划中的 #4346 (comment)

不用X换用随机ASCII字符的方法非常简单 只需要确定字符集(eg base62)然后计算它在哈夫曼编码下的压缩率然后将预期的xpadding长度除以它做修正就行了 最后的结果会趋近xpadding的长度而且还会额外附加一点随机性 这早在全部用X填充之前我就想到了 不过问题不大而且这个字母占8个位确实很有缘所以当初就没提这一点(当然复杂一点的话精确长度也是完全可以做到的)

I thought the pr was in english, and you are replying in chinese. where is your conscience or respect? what were you thinking when making this reply and then closing this pr?

and then, you did not give an exact valid reason on why you closed this pr. all you said is that you can handle yourself the random generation of something I still don't understand. you ignored the main purpose of this pr, that is clearly in it's title:

XHTTP: rename X-Padding to X-Signature, add xPaddingQueryParam

..or if you can't read english, here you go:

XHTTP：将X-Padding重命名为X-Signature，新增xPaddingQueryParam参数

the #4346 issue you pointed to is dead. nothing was made or changed since (1 month) the last reply in it (not counting mine reply)

[hxehex]

also, the xray phrase "Xray, Penetrates Everything." right now is invalid - because it does not "penetrate" CDNvideo

[RPRX]

我们看得懂英文，回复用中文纯粹是因为对我们来说码字速度更快，没有别的意思

我觉得从你改了多个部分的代码以及“Further improvements”的思考来看，你对 Xray 的架构有不浅的研究、完全理解 XHTTP 的工作原理、且有发散性的思维，也就是说你是一个持续改进 XHTTP 的不错的人选，所以我会 reopen 这个 PR，只是怎么改还需讨论

就像 Vision Seed 一样，本来在等 GFW 动手，如果 XHTTP 再不加自定义的话等下隔壁出个类似的传输层号称啥都能改，绷不住

Renamed X-Padding header to X-Signature - a lot more common and widely used by actual CDN users.

修改默认名称会破坏 XHTTP 新旧版本之间的兼容性，应当加选项

Added xPaddingQueryParam. It can be set in the config to change the name of the query param used to pass the padding data. The default is x_padding
Changed the way the padding data is generated. It now consists of chars X and Z, not only X.

然后关于这两点我思考过 #4346 (comment) ，如果要改应当加上七八个选项，或者就干脆不带 padding，你觉得哪一种更好？

[RPRX]

also, the xray phrase "Xray, Penetrates Everything." right now is invalid - because it does not "penetrate" CDNvideo

我突然觉得 Xray 这名字挺贴切的，对不同墙的穿透性不同，但毕竟不是 Gamma-ray，尽力而为，如果完全断网那只能星链

[hxehex]

also, the xray phrase "Xray, Penetrates Everything." right now is invalid - because it does not "penetrate" CDNvideo

我突然觉得 Xray 这名字挺贴切的，对不同墙的穿透性不同，但毕竟不是 Gamma-ray，尽力而为，如果完全断网那只能星链

🫤 starlink is not available nor it does work in Russia (officially)

[hxehex]

修改默认名称会破坏 XHTTP 新旧版本之间的兼容性，应当加选项

I guess neither me or @paqx thought about that

I think maybe we should revert that one change but still allow customizing the header name

[Fangliding]

这个部分暂时不需要pr 不过它是计划中的 #4346 (comment)
不用X换用随机ASCII字符的方法非常简单 只需要确定字符集(eg base62)然后计算它在哈夫曼编码下的压缩率然后将预期的xpadding长度除以它做修正就行了 最后的结果会趋近xpadding的长度而且还会额外附加一点随机性 这早在全部用X填充之前我就想到了 不过问题不大而且这个字母占8个位确实很有缘所以当初就没提这一点(当然复杂一点的话精确长度也是完全可以做到的)

I thought the pr was in english, and you are replying in chinese. where is your conscience or respect? what were you thinking when making this reply and then closing this pr?

and then, you did not give an exact valid reason on why you closed this pr. all you said is that you can handle yourself the random generation of something I still don't understand. you ignored the main purpose of this pr, that is clearly in it's title:

XHTTP: rename X-Padding to X-Signature, add xPaddingQueryParam

..or if you can't read english, here you go:

XHTTP：将X-Padding重命名为X-Signature，新增xPaddingQueryParam参数

the #4346 issue you pointed to is dead. nothing was made or changed since (1 month) the last reply in it (not counting mine reply)

我完全看得懂英文 反正很多人都要借助翻译软件译成自己的母语我还不如也直接敲中文尽可能保留原始意思
我打一开始就看了这些更改没什么很大的建设性 重命名header的名字和把里面填充的字母从全X改成XZ混合并没有什么意义 要检测也就改几行规则的事情 改名字还顺便破坏兼容性 但是这个header确实打算开放修改 我才贴了那条link

[hxehex]

@paqx will reply about this problem a bit later he can't answer right now

[paqx]

@RPRX

Thank you for reopening the PR. Russian users really need these changes at the moment to bypass censorship.

Based on your modifications to multiple parts of the code and your consideration of "Further improvements," I believe you have a deep understanding of Xray's architecture, a complete grasp of XHTTP's working principles, and divergent thinking. In other words, you are a good candidate for continuous improvement of XHTTP. Therefore, I will reopen this PR; however, how to modify it still needs discussion.

Thanks, but actually I do web developing in PHP/Python + DevOps. Yesterday was the first day I started to dig the xray code and I haven't coded in the go programming language before. Therefore, I hope you will help me implement the things I outlined in the first post.

Changing the default name will break compatibility between new and old versions of XHTTP; an option should be added.

I will return X-Padding as the default header name for backward compatibility and try to make it possible to change the header name via the config file.

Then I thought about these two points #4346 (comment) . If we were to change it, we should add seven or eight options, or just remove the padding altogether. Which do you think is better?

Could you please explain the idea behind the seven or eight options? In my PR, I implemented the possibility to set any query param name instead of x_padding. Doesn't this allow users to set any name and not chose from just 7-8 options? Or do you mean something else?

As I mentioned, the xray code is new to me. I don't yet understand why some decisions were made. I gathered my knowledge about XHTTP mainly from this post. My current understanding is that the x_padding query param as well as the X-Padding header were added to make the size of HTTP requests more random (so that censors cannot easily detect HTTP requests of the same length and block them). If that's true, I don't think it's a good idea to completely remove the padding. I realize that the primary goal of xray is to bypass censors and not CDNs. However, CDNs in Russia decided to cooperate with censors so I think making xray less detectable for CDNs is necessary too (but not by making it less stealthy for censors).

@Fangliding

Renaming the header and changing the padding from all Xs to a mix of XZs is meaningless. Detection would only require changing a few lines of rules. Renaming it also breaks compatibility. But this header is indeed intended to be open for modification, which is why I posted that link.

I agree that my current changes do dot provide enough protection but I needed to start from something. It currently works and allowed me to bypass the CDNVideo filters. They simply added some regex that searches for x_padding query param followed by a sequence of Xs.

I only changed Xs to a sequence of X+Z because this post specifically mentions that it was chosen since it gets compressed into 8 bits and it sounded like it's important for xray operation. And the only suitable letter that gets compressed into the same 8 bits is Z. So I used what I could use. It would great if you could explain why the padding data is generated like a sequence of Xs now.

Obviously if CDNVideo notice some other param followed by a sequence of Z+X, they will easily block it again. But if I manage to change the way the padding data is generated and make it look random like some hash, it will be a lot harder for them.

For example, real CDN users can often use params like _dc for cache busting and if the padding looks something like this:

_dc=iubwefwe2423bfwebnklewb2b23t23

I doubt they will decide to block it because it might cause serious issues for many real users. Isn't it what XHTTP is all about?

The method of using random ASCII characters instead of 'X' is very simple. You just need to determine the character set (e.g., base62), calculate its compression ratio under Huffman coding, and then divide the expected xpadding length by this ratio to make the adjustment. The final result will approximate the xpadding length and also add a bit of randomness. I thought of this before using all 'X's for padding, but it wasn't a big problem, and the fact that this character occupies 8 bits was quite coincidental, so I didn't mention it at the time (of course, a more complex method could achieve the exact length).

I will try to implement this in the PR.

Thanks.

[gfw-killer]

I think the new options could be

1. the padding position (query-string / header / cookie / multipart)
2. it's key-name
3. padding character-set (repeated X or random A-Z,a-z,0-9)
4. upload method (POST / PUT / PATCH)
   it is also possible to upload data using GET request but not all webservers supports it, same about OPTIONS/TRACE/CONNECT/HEAD
   There is also 'Resumable Upload APIs' like Tus; https://developers.cloudflare.com/stream/uploading-videos/resumable-uploads/ (with no need to gRPC support))

[gfw-killer]

I remember there was another obvious characteristic too
POST /yourpath/sameUUID/seq
the connection uuid could also have different position and forms, and also the seq

[paqx]

Hi everyone!

I just finished the new padding generation method and also added capability to pass and receive padding in different places. At the same time I tried to preserve backward compatibility.

The new parameters that I added are:

1. xPaddingObfsMode
2. xPaddingKey
3. xPaddingHeader
4. xPaddingPlacement
5. xPaddingMethod

xPaddingObfsMode - for backward compatibility. If's set to false (the default value), the padding will be generated and placed like before.

xPaddingKey - the name of the key for storing the padding value. Depending on the xPaddingPlacement, it can be:

1. name of the query param placed inside of a URL which in turn is placed inside of an HTTP header
2. name of the cookie
3. name of the HTTP header
4. name of the query parameter

multipart (I understand it as padding in the request/response body) seemed difficult for me, so I decided to skip it for now.

xPaddingHeader - the name of the header. Only makes sense when padding is passed as a header value or a query inside of a header value.

xPaddingPlacement - only works when xPaddingObfsMode=true. Defines a place to put the padding (queryInHeader, cookie, header, query).

xPaddingMethod - how to generate padding: repeat-x (like now) or tokenish. For tokenish I did some testing and found out that Huffman encoding gives ~20% size reduction for base62 sequences. So I generate a sequence taking into account this compression ratio and then try to adjust it until the validation tolerance (currently 2 bytes) is reached.

Files

• transport/internet/splithttp/browser_client.go
• transport/internet/splithttp/client.go
• transport/internet/splithttp/hub.go
  Have easy access to the padding config.

However, files

• app/dns/nameserver_doh.go
• transport/internet/tls/ech.go
  don't. And I just don't know how to pass the padding configs there. So far I simply removed padding from them and my question is whether it's fine. If not, could you suggest better ideas?

I did some testing and the new features seem to work. But it would be great to have the code reviewed.

In the meantime, I will try to implement more upload methods besides POST.

Thanks.

[Fangliding]

你可以用这个utlils函数替换其他两个地方的padding（说实话因为header是写死的它们意义不大 可以保持X 不知道为什么要删掉。。）

[paqx]

@Fangliding

Thanks for the comment. If they are not very meaningful, let's revert the changes. I used the helper function that you added and returned the padding to app/dns/nameserver_doh.go and transport/internet/tls/ech.go.

I also added another parameter. Now it's possible to specify the uplink HTTP method using uplinkHTTPMethod.

Currently, some Russian CDNs block xray by simply disabling the POST method, but other methods usually work. One of such CDNs is Yandex CDN (and their partner EdgeCDN). Using the new uplinkHTTPMethod I switched to the PUT method and it worked. I also tested the HEAD method and it worked too.

Now I would like to change the user agent in requests to something that looks more like a real user and maybe do something about the UUID and seq in each request URL.

[Fangliding]

超级大量的put和head请求比post更加奇怪 本质还是猫鼠游戏

[paqx]

I can see that Russian CDNs are blocking xray with care. Examples:

• CDNVideo used a precise regex that matches x_padding=XXXX. Simple changing X to a combination of X+Z bypasses the filter.
• Other CDNs like Yandex only disabled the POST requests, but not PUT, PATCH, TRACE, etc. I don't have any statistics, but as a web developer I haven't seen any websites use CDNs with methods other than GET. Even POST looks unusual to me. In my opinion, they could just leave only the GET method and disable all others in advance. And yet they didn't do it.

Why? My guess is that real CDN customers might have special needs and may actually use methods other then POST in their web apps. And CDNs can't extend their filters forever. Because at some point it might severely impact their real paying customers.

So I still think the changes make sense.

As a last resort, I thought about using GET requests for uplink (via query or headers). I guess it will be super-slow, but it's better than nothing.

[hxehex]

超级大量的put和head请求比post更加奇怪 本质还是猫鼠游戏

wdym

[gfw-killer]

本质还是猫鼠游戏

All this proxy stuff is a cat-and-mouse game, GFW use JA3, proxy devs use uTLS, they find vulnerability to detect uTLS, proxy devs find and patch or invent things Naiveproxy, GFW continues and proxy devs continue too
The difference is only, that when there is no TLS, it's harder, but still not impossible, raising the number of false-positives will do the job

Also a reminder that when user use a CDN that is made by the local government, there must be option to disable the padding completely, as the CDN is literally whitelisted by the local GFW
it's more important to have options to change the connection uuid and seq positions, and even multiple forms with different lengths for the uuid (POST /yourpath/sameUUID/seq)

As a last resort, I thought about using GET requests for uplink (via query or headers). I guess it will be super-slow, but it's better than nothing.

Performance will not be very bad, because upload traffic is low in normal web browsing, except when user uploads a file to somewhere
And your local CDN may support a huge max-header-size, or maybe they set a better size limit on cookies or query-string

Check this one too (I think most CDNs will not support it, but your local CDN may do)
https://www.baeldung.com/cs/http-get-with-body

[Fangliding]

All this proxy stuff is a cat-and-mouse game, GFW use JA3, proxy devs use uTLS, they find vulnerability to detect uTLS, proxy devs find and patch or invent things Naiveproxy, GFW continues and proxy devs continue too
The difference is only, that when there is no TLS, it's harder, but still not impossible, raising the number of false-positives will do the job

那能一样吗 用utls是为了模仿流量更常见的client hello 这用更少见其他方法代替post 不是弄反了?

[gfw-killer]

My view is that having many options will increase the count of false-positives for them and is beneficial
Currently detecting XHTTP does not even need a RegEx, this options will help a lot
If the author make no breaking change and keep the design minimal and clean, it will bother no one

[nedogimov-m]

Check this one too (I think most CDNs will not support it, but your local CDN may do) https://www.baeldung.com/cs/http-get-with-body

A lot of Russian CDNs support this

[yuhan6665]

Thank @paqx for your work!
For some reason the reverse test cases are hanging..

[paqx]

@yuhan6665 Hi! Thanks for the information. I'll check why it happens when I finish the remaining features I would like to add .

[paqx]

I added parameters to change the placement and keys of session Id (uuid) and seq (POST /yourpath/sameUUID/seq). By default, they will be placed in the path for backward compatibility.

• sessionPlacement - can be path, query, cookie or header.
• sessionKey - the name of the key (not applicable for path)
• seqPlacement - can be path, query, cookie or header. If sessionPlacement is path, seqPlacement must also be path.
• seqKey - the name of the key (not applicable for path)

The last thing I would like to implement in this PR is the ability to use the GET method for uplink transmission and also pass the data in the query, header or body instead of the request body.

[RPRX]

我其实懒得去设计哪些地方应该怎么改，要不直接在 XHTTP 内允许多人 PR 多种变体得了，只要两端都是 Xray-core 时能连上就行

[RPRX]

然后看哪个变体比较能经过时间的检验，最终会被纳入 XHTTP 官方版的设计，非 Xray 的实现也可以同步过去了