please help: multi task download /u/sdyxz Python Education

I’v been working on this code for days, trying to download okx crypto market candle data. The server has some limits:

1, rate limit: 40 requests max / 2 seconds.

2, pagination: for each symbol, could only download 1440 candles at most, and have to use pagination since each request has 300 candles max limit.

This code can deal with pagination correctly in fetch(). And I tried to make multi thread download as fast as possible while meeting rate limit, in fetchall().

I guess the problem is: using pagination, each symbol’s one request turned into several requests, thus sabotaged my effort of meeting rate limit? Now I got many ‘429, {“msg”:”Too Many Requests”,”code”:”50011″}’ error for many symbols, but not all.

I tried lowering max_requests value from 40 to 20 or 25, it can improve, but no guaranty that it will have no error, considering in a different environment or different time. I want a reasonable/scientific solution.

I also tried give it more tries if a symbol fetch got 429, but maybe more tries means sabotage rate limit even more?

Could anybody please help me fix this?

code is here

I’v been working on this code for days, trying to download okx crypto market candle data. The server has some limits:

1, rate limit: 40 requests max / 2 seconds.

2, pagination: for each symbol, could only download 1440 candles at most, and have to use pagination since each request has 300 candles max limit.

This code can deal with pagination correctly in fetch(). And I tried to make multi thread download as fast as possible while meeting rate limit, in fetchall().

I guess the problem is: using pagination, each symbol’s one request turned into several requests, thus sabotaged my effort of meeting rate limit? Now I got many ‘429, {“msg”:”Too Many Requests”,”code”:”50011″}’ error for many symbols, but not all.

I tried lowering max_requests value from 40 to 20 or 25, it can improve, but no guaranty that it will have no error, considering in a different environment or different time. I want a reasonable/scientific solution.

I also tried give it more tries if a symbol fetch got 429, but maybe more tries means sabotage rate limit even more?

Could anybody please help me fix this?

code is here