usage of proxies is broken in nightly release (net::ERR_FAILED)

[richie256]

Expected Behavior

Using proxies with the docker nightly build or nodejs (using the latest code from github) should works fine without net::ERR_FAILED error.

Current Behavior

Using proxies with the nightly build or nodejs (using the latest code from github) return net::ERR_FAILED error. It works as expected using latest docker image.

Steps to Reproduce

1. Configure the dotenv and global.proxies file with valid proxies
2. Launch docker run -it --rm --env-file ./dotenv -v $(pwd)/global.proxies:/app/global.proxies ghcr.io/jef/streetmerchant:nightly

OR

1. Clone the streetmerchant github.
2. Configure the dotenv and global.proxies file with valid proxies
3. Install and run: npm run start

Environment

• OS: I've tried Windows 10 and MacOS.
• NodeJS: I've tried NodeJS 16, and NodeJS 15.

dotenv

Nothing Particular dotenv. Any should reproduce the same result.

global.proxies

socks5://<my_username>:<my_password>@<my_secret_ip_address_1>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_2>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_3>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_4>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_5>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_6>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_7>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_8>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_9>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_10>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_11>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_12>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_13>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_14>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_15>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_16>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_17>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_18>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_19>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_20>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_21>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_22>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_23>:1085

Logs

Using proxies with the nightly build we got net::ERR_FAILED error.

[1:38:42 AM] error :: ✖ [2/23] [newegg-ca] evga 3080 ftw3 - net::ERR_FAILED at https://www.newegg.ca/evga-geforce-rtx-3080-10g-p5-3895-kr/p/N82E16814487519
[1:38:42 AM] error :: ✖ [3/23] [canadacomputers] msi 3080 gaming x trio - net::ERR_FAILED at https://www.canadacomputers.com/product_info.php?cPath=43_557_559&item_id=181348&language=en
[1:38:42 AM] error :: ✖ [4/23] [memoryexpress] evga 3080 xc3 black - net::ERR_FAILED at https://www.memoryexpress.com/Products/MX00114094

Exactly the same config works as expected with the latest docker image

[1:47:52 AM] info :: ✖ [2/23] [memoryexpress] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[1:47:52 AM] info :: ✖ [1/23] [canadacomputers] [evga (3080)] ftw3 ultra :: OUT OF STOCK

[jef]

Hi there, thanks for reporting. Is this still the case?

[richie256]

hello @jef ,

Yes it is still Broken.

I've done some research: I've list all the commit between HEAD and Tag v3.5.0 and tested the versions.
Based on my test, it doesn't work since the commit: 1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies

Proxy Test Results

00b468a73832b959e0c6ac1ed67b0b23856db76b (HEAD -> main, origin/main, origin/HEAD) docs: make templates required
74d71e457956743566796404458bd1d5d84598db docs: update issue templates
9aa3f164a59cd521701f55bfe4a54872b1c5fc4d docs: update issue templates
71711131ff4c57abe3016e543b7d0ce994e0ecd0 docs: update issue templates
9c346e70fac17854d347dfbdbc238bfb0056c6a8 fix(terraform): update IAM configuration for ECS deployment (#2675)
61af2a4325733463c918396868d313b7632bb310 fix: partial url regexp (#2382)
f14a1ca76e41943e639e08a9c8a6299dfa362d9d chore: update dependencies       ==> TEST: FAILED
0c589a71f259c80ff21a4bb623f9316cb7702a61 fix(bestbuy-ca): avoid false positive (#2694)
0b060e8f6aca1793d04474210aeec98212172014 feat: add a few stores for spain (#2646)
4bb58ac26192354ccbcd0c416aba78df2234cf19 chore(coolblue): add ps5 consoles (#2597)
e0303106f725ee9726c0b7e2f29d97fe5137ca90 chore(deps): bump actions/stale from 3 to 4 (#2671) ==> TEST: FAILED
5e9b3db6f1630f2000da7f9a8b0dd0dc34c8341f chore: revert puppeteer, run linter
1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies	 ==> TEST: FAILED
4e3eadea083071eba563dffc70d749db334dce69 chore: bump node 16.5.0 	 ==> TEST: OK
8c7b8aabe4633757dea836b7d11af2a407869e90 feat: add alternate-fr, 3070ti, change alternate to alternate.de (#2653). ==> TEST: OK
c0f46d77cf9d4a3f7481aec5b0ddf33883b47d6e feat(stores-nz): update labels (#2486)
cc0aaeca77dd18a2c9797055f26edf6935fa755b chore(bestbuy): add 3080ti models (#2670)
b33f678090396b77a5765115d7a3fa42f6de06c8 chore: prune invalid links (#2592)
51c9b3ab2ac251c9c3abebf64025e1a03ecf4944 feat: add 3080ti (#2600)
4cfbd37ae0d28d1c2b8cf596b5dcb941daa7d9bc feat: add 3080ti (#2598)
52b5e5e79ac381b585ec1dd63288aefe4f9da9d1 ci: fix nightly
812dc5ea8d9830c251a526c02de05a15bfe20602 ci: change check-date name
3b43732f291d263792585fa05264dcf322b55620 chore: update dependencies and nightly workflow
e4a267f675800b04d06ddb4212de1373a767a113 chore: update stale workflow
da635f41dd3c3e949f3d77fc2b1bcbe860639179 build: remove smartthings
2e5b07c93fb02677596eb5a2f2ea857b99c2d4d9 chore(deps): bump nodemailer from 6.6.0 to 6.6.1 (#2568)
a751cb11c53c7cf6b7883e69eb5e8e2fe1e57adf chore(deps): bump node-notifier from 9.0.1 to 10.0.0 (#2583)
e4ebdc4719afdf390f7060bbc755229dcaba1410 chore(deps-dev): bump @types/cheerio from 0.22.28 to 0.22.29 (#2580)
1cb1787c27b2e7e37e3e989be686b635f98442e9 chore(deps-dev): bump @types/nodemailer from 6.4.1 to 6.4.2 (#2582)
9fc5ec06ff92e458ea9649f22af9df1ae5ba1d95 chore(deps-dev): bump webpack from 5.37.0 to 5.38.0 (#2579)  ==> TEST: OK
9ba964a587158b551d7217764eb9799073aea21f chore: bump node version
9e13cd7dbb8026c53da4499309ebef03d51c3e92 chore(deps-dev): bump webpack from 5.37.0 to 5.37.1 (#2558)
3e74a4c7c41c939b9ce0952e089f36269f3fb46f chore(deps-dev): bump @types/node from 15.3.0 to 15.6.1 (#2571)
7d35c2b8f0620501061220baefeb499d6e66e2da chore(deps): bump twilio from 3.62.0 to 3.63.0 (#2561)
36e188fd36d31a915d9ac59016710b6f3fa4436d chore(deps): bump open from 8.0.2 to 8.2.0 (#2572)
11220750143df7ebdc011aafccb8e035bfd2bc5d chore(alternate): add PS5 bundle (#2562)
8854ce97bba7858dfc9385530e0c090200944ede fix: clear cookies when rotating proxies (#2575)
ebf8b20071dc628b1f47b96ee2b03f888c056dd9 ci: fix documentation
729afd0ef3f4ceea1ae38b04e1a25bc4c4dcd3ec chore(deps): bump top-user-agents from 1.0.25 to 1.0.26 (#2556)
42a24a633c537941d63a6b76ed9598179dda9ea4 chore(computeruniverse): add backoff code (#2542)
32fe440049ce97c5683ca03276212899b2211af5 feat(gotify): add priority option (#2546)
0378a7cf0aa673933af35c3a3605f147615f6e96 chore(netonnet-no): update selectors (#2547)
1e62878c8d1b3c0b49189b05f846d555cfc205b3 chore(deps): bump node 16.1.0
9f58c22c0c6d14d0f0e59782bbe71a18b8329f6f style(gotify): linting
d12639ee8561240d99ef5c2c364255ff808f78b0 chore(notification): add better logging, remove debug
8f2de5dc68667f6b9f0033bed129fd6cdbd9d8a7 feat(notification): add gotify
1ab0af234ca76b5db8602ffe04a1fc80b3c6515b chore(deps): bump dependencies
6e614a24918f38d189a3d3b70174fbd1590f78cc feat(captcha): add support for configurable capture type (#2468)
b6af582f70843c5608249c5d629fdec8f10630dd chore: remove hooks
d0a6c7e85950fe3cffbcb30662967a687aaf7347 chore(otto): add labels for xbox series x & s (#2515)
c9049a3054d35f8a37340e552e19841e3ecaf631 chore(elkjop): update selectors (#2505)
64120e73d8f3a01dd5852044ffa0ef6b0ce31730 chore(mediamarkt): update captcha selector (#2496)
4d8930616d2296fcdd6c0d79bac2109f1388a0b0 chore(xbox): update oos selector
164902a587d80f0421b44079e0616ba37a7a743f fix(captcha): use Slack captcha handler token when provided (#2460)
790fe9b6cf4f9627e7f3dc7acc572737a739e4cc fix(amazon): false positives for xbox (#2497)
7c4c6ed79ecd89767ffbb3577716be28ce6c1bdc chore(multicom): update selectors (#2499)
e397306f1a3b1576c0595dec5fae7be277e88f25 chore(deps-dev): bump webpack from 5.35.0 to 5.36.2 (#2480)
b57905291df85ee2f722e5e3a81d1cecd0ab6ee4 feat: add INCOGNITO to dotenv-example and volta support (#2484)
441617f6729ab61174f7bcfe287c41a34a78728f feat(store): add amd-at (#2462)
5a744b1add24dd4d3c250f41382a034212d9c5b0 feat(store): add cyberport-at (#2463)
eb76f0d2b78726c4380853bd47edcf9b1302e7bb fix: remove spaces from price (#2482)
41305f1ed5a81240ba7aa0dda1619d635d917371 feat(store): add multicom.no and more products to NO stores (#2483)
39fac90ba4feadb7e5c681db60e60a83749f5e62 fix(lookup): prevent terminal from being hidden (#2461)
cac7c20193caaec5d41291104e66aa01a65adceb feat: add waitUntil and waitForSelctor to linksBuilder (#2437)
35f9eb43b6cca8a92f60ba2870e60a225b4c1da6 fix(alternate): use button selector instead of text (#2419)
b6ac6c53ddc34ac35b537dd99415bbcb9eec0de7 fix(nodemon): use npx with ts-node (#2423)
092919ab349edac6a8b5b4747917a1a232235da7 fix(microcenter): add outOfStock label (#2432)
d652b4936435377d36268141a2b63c8798f7f95b refactor(captcha): use seconds instead of ms (#2424)
2af153192c2cf3ac2bf9ba0d1d3201f250363c9b chore: change urls.  ==> TEST: OK
5937d0141d1c34fdb90475bffbbe32a2977c789b chore(alternate): add new oos label (#2409)
3c00eca22378afb2a18aa76c1d4204107e6364c2 chore: bump dependencies (#2413)
409769e6dbd3add614fcd985731bb244297a2c2d feat: add interactive captcha handler with Slack and Discord (#2385)
8c39d0d30a9f8db7eda8665eb08d350fadf157e4 chore(materiel): add 3070 and 3080 cards (#2393)
4eb702b9a1187c8dcf159ea991ab1049c3c44bb2 fix(amazon-es): use euroFormat (#2371)
2be33ac6147f8f7460d108f361d954b17a335bd9 fix(web): add favicon to prevent missing icon console error (#2375)
d4dc397e9a848d6d2ba1ed56188fb0ddda98e795 chore(store): add some PS5 bundles (#2021)
d13f7f1ff91ada223e8ce7eb50d9a80030cbe668 fix(docker): disable opening browser
f0e5c5d17bad282480da30cc68e5bd8eb9d80844 chore: update pr template
6e9b1a9640ca93f620623d53624230514fac6d40 feat(store): add amd-be store (#2338)
49c9e8c0dc5c3b25ce2aa20265ccd2865f229385 feat(store): add amd-nl store (#2072)
bc59da11d512f6f55449781cb8f7f4bd1d3e7e57 fix(discord): properly wait for webhooks (#2352)
aed9cfe98919e6524bfde0156a308c9a56d4dfc0 chore(deps): bump dependencies
e2d804027a3ec6a19201ef9ce35dcb4c4dee7025 feat(store): add MightyApe, NoelLeeming and TheWarehouse (NZ) (#2133)
c1c92bd25b7b744a05f21664f2029a0cb0683d35 chore(gamestop): update inventory (#2239)
ae8e72a73044b8311add15d2b5c10936bae16e82 feat(store): add mediamarkt austria (#1836)
211e3e2692c1c97f607b39b73623a09958c69d81 chore(currys): add rx6800, rx6800xt and rx6900xt (#2236)
7ec8a160cf9844df34578fe9f413b88d6141178c refactor(materiel): update docs, change name
0ebe7451e4e4c6e46c64e3782bd51a92ce594dca feat(store): add materiel.net (french), 35 3060ti references (#2287)
5b2fe88a83f17ff98ebe5c91806c7eb5c1dc3bdc chore(amazon-fr): update evga 3080 ftw3 link (#2286)
bf8f242f4b33796fcee91a863f0e086fcec29537 refactor(freemobile): remove local throw
6be2e0d491646d6e88f72f67c694874b11cda03a feat(notification): add support for Free mobile (#2285)
a25b7f19cc8e403ffd7856e631cde7e9df63f1cc refactor: simplify maxPrice logic
e006c48795b8fa104f0cee5584517a5f164d6512 chore: change verbiage to be more generic
9b02c36370aaf7fef20a90b2427273c2f7b30f00 feat: change max price behavior
b4121463ed7f25566cfa4ff508fa0c4d03237a9d ci: update stale labels
d03aea05a7783c9cc6a18c8baf0381050c105357 feat(pushover): ability to specify device (#2101)
ea095a364c250f805704c1189c0f3e697e09673d chore(alternate): update links and selectors (#2282)
c7e24e77d415819d27043d5814b2133b46549e57 chore(amd): add rx6800xt midnight black (#2321)
3c1419f170b1d00aa6ad08ad89752fc8f9134a93 chore: bump stale workflow to 60 days
cd6e1f614bcaf00bc468c32a1b5a98cb51c7730f fix: add missing tests in Dockerfile (#2323)
91a0ff5e1ba83efabe5ef45c6a3e17e1c564f16b chore(deps): bump dependencies, node 15.14.0
bbfa808b820ee2d16c65dfdcecce58645f4899d0 feat: color empty stores (#2312)
c2c8531ea26e2b75f50ed384f5128e5df54053e9 docs: update badges
113c9ce19f3ed1f140eec7cd555ec373f25d256f feat(series): support for rx6700xt (#2290)
ba35706cca84a425da8d5a5bd6b86dbecbadffa1 docs: update Xbox Series S and X price limit desc (#2098)
2ebbf55004da325c38c1eb12bcd5dc3264791604 chore(deps): bump node from 15.11.0-alpine3.13 to 15.12.0-alpine3.13 (#2185)
1714e89af7964d06b01120fb75595ef8ec8177ba feat(store): Add AO and BT (UK) (#2089)
983cc525164744e4d3bff1475d12b5563d23200e chore(amazon-uk): add 3060 cards (#2229)
a7a5e6954b3568b40d360982d5f11287c1b7f440 chore(deps-dev): bump @types/node from 14.14.35 to 14.14.36 (#2233)
2a1734a71d20e9a65c04edcd4318eaa3b5872a7d chore(deps): bump dependencies
83ce922761f85a13424ed43a1961668ffc870485 feat(pushover): add support for sound (#2203)
0a9ce0dd4c7681cba7681d570db44f65359d83cf chore(store): add Nvidia RTX Cards (#2204)
b2ce7157d9631d6edd45debcb9ab85bc79241133 chore(walmart): update 5900x url (#2184)
b0efc837bc554f5c8635e125b67775d8b1f8f6b3 fix(playstation): fix inStock selector (#2187)
54883f0ad7cf86052035d88ed2ffbc612c439725 chore(deps): bump open from 8.0.2 to 8.0.3 (#2186)
4bd048ff7a496e82e66293a9706581b8d10af6b3 chore: add APNS to dotenv-example (#2174)
99a274dbea088a6bcb053b1c1bf7812d0796442a fix(apns): use when specified
10f7e0cf5d8f633aa0195a1f9403965b4347e017 chore: bump dependencies
2775124d80f86466dbc0c3bcb227f489f35aab62 chore(deps-dev): bump webpack from 5.24.2 to 5.26.3 (#2164)
a575143b244525ff87579b689ed90bf6ae251847 chore(deps-dev): bump mocha from 8.3.0 to 8.3.2 (#2167)
4cdda9890cb30443826e6df02b99d41dbbea026d chore(deps-dev): bump @types/sinon from 9.0.10 to 9.0.11 (#2166)
53e7a8d692b76155699b371e926a2b9687d4a081 chore: remove unused dependencies
b516e268428219de70358821683e2dd74df59943 chore(deps-dev): bump typescript from 4.2.2 to 4.2.3 (#2092)
00e4ddce162877a4a08747955859508f5af0a802 chore(deps): bump open from 7.4.2 to 8.0.2 (#2078)
eeab4b112a6482d4fd381014e44fd0d6e071c494 chore(deps): bump twitch-chat-client from 4.4.13 to 4.5.1 (#2127)
0ff8158511f4f90dc26407fffa22ab0a4ad3aa0b revert: replace euroFormat with regex (#2131)
91c4f12308905f4e6d65341ba54ae814c8ad94d6 revert: parsing prices with thousands separator and European format (#2153)
6b78b2ee4445ad2b868b6f848730a474ca12d657 feat(notifications): add apple push notifications (#2077)
d46c6366b70ca668437bd2aab87c3c529f4c569b chore(microcenter): add 3090 cards
7fcce586bf03a573a1bba143c7d56e046418d31f chore(amd): update rx6900xt urls (#2094)
245d110d0a52df8b2d05f020aaaa1bfa8be7cab4 chore(asus): update urls (#2155)
61f8e3a9a03136682035881fce6d14356717d1ac fix: parsing prices with thousands separator and European format (#2153)
795c7230ad4632c81b17f5c2aeb3b0e88b2b04e3 refactor(store): replace euroFormat with regex (#2131)
afe7a709009f43fba1cef22aa6c136976507c7bc chore(nodejs): bump version (#2088)
f65aab87dade27c9f271ca816f83fe5a57ff7da6 docs: update node references
c2a22716cce0de6cbdbfdf7ebdceb97844a22e0f ci: fix docker release	   ==> TEST: OK

thanks!

[jef]

I reverted that puppeteer change, which may have done some damage. Do you mind testing with this commit?

5e9b3db chore: revert puppeteer, run linter

And to clarify, it works with the commit just prior to the one you posted?

4e3eade chore: bump node 16.5.0

[richie256]

Hello @jef ,

And to clarify, it works with the commit just prior to the one you posted?

Exactly, it work with the commit just prior the one posted:

• I got net::ERR_FAILED with (1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies) or any newer commit.
• Works fine with commit just before: 4e3eadea083071eba563dffc70d749db334dce69 chore: bump node 16.5.0

Same failure net::ERR_FAILED with the docker nightly build, as of now.

docker pull ghcr.io/jef/streetmerchant:nightly
docker run -it --rm --env-file ./dotenv -v $(pwd)/global.proxies:/app/global.proxies ghcr.io/jef/streetmerchant:nightly

[richie256]

@jef

Do you mind testing with this commit?

5e9b3db chore: revert puppeteer, run linter

This fails net::ERR_FAILED as well using proxies.

[Fmstrat]

If it's of help, I had to roll back to https://github.com/jef/streetmerchant/pkgs/container/streetmerchant/3431703 to get things working.

[Tiippex]

Oh, I came across the same issue. It is present in 0906f64 (nightly).

I'm happy to help :) Anything we can do here to help you solve the issue?

[andrewmackrodt]

I'm not sure if this is still an issue, but I had to change the following in my fork which may be helpful:

• If the PROXY_ADDRESS configured proxy requires credentials, they cannot be passed via --proxy-server=. They must be passed using page.authenticate.
• @doridian/puppeteer-page-proxy was not working for me, I am using puppeteer latest so maybe this doesn't repro in upstream. Switching the library out for puppeteer-proxy works for proxies set via proxyList.

[adamscybot]

@andrewmackrodt I was just about to set off on everything you have already achieved in your fork, amazing.

There's an issue where if it it hits an issue fetching links, it may die since it tries to abort an already failed req. Im not quite sure why given the abort call has error handling, I guess there's some uncaught exception elsewhere in the event loop of pupeteer.

Though the core problem I have is failing reqs in the first place, as my residential proxies seem to just be triggering all sorts of IP blocking/cloudflare on certain UK retailers 👀.

Dont suppose you'd be willing to share what proxy service you've had success with?

Edit: It would help if I picked proxies from the right country 🤦-- that improved things. Very slow though and I don't have much confidence. Headless also seems to guarantee to trigger something or other.

The errors are thrashing though and with many stores they kill all the tabs I think and it reopens. Hard to tell whats going on:

[19:43:41] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Request is already handled! {
  "stack": "Error: Request is already handled!\n    at assert (\\node_modules\\.pnpm\\puppeteer-core@24.1.1\\node_modules\\puppeteer-core\\lib\\cjs\\puppeteer\\util\\assert.js:18:15)\n    at CdpHTTPRequest.abort (\\node_modules\\.pnpm\\puppeteer-core@24.1.1\\node_modules\\puppeteer-core\\lib\\cjs\\puppeteer\\api\\HTTPRequest.js:306:32)\n    at proxyRequest0 (\\node_modules\\.pnpm\\puppeteer-proxy@1.0.3\\node_modules\\puppeteer-proxy\\dist\\routines\\proxyRequest.js:136:39)\n    at async handleDefaultContextProxy (\\build\\src\\proxy.js:76:9)\n    at async \\build\\src\\util.js:115:21\n    at async CdpHTTPRequest.finalizeInterceptions (\\node_modules\\.pnpm\\puppeteer-core@24.1.1\\node_modules\\puppeteer-core\\lib\\cjs\\puppeteer\\api\\HTTPRequest.js:151:9)"

[andrewmackrodt]

@adamscybot it's working for these retailers with headless+incognito modes, using a round-robin proxy for each request: awd,box,ebuyer,novatech,nvidia-gb,overclockers,scan. I'm using Oxylabs ISP proxies.

currys is trickier; fetching links works with the above setup but the actual product pages trigger cloudflare and I've only gotten that working with default context and headed mode (manual cloudflare verification). And even then, it only works some of the time.

I'll open the issues section in my fork so as not to pollute this discussion.