question

Upvotes
Accepted
1 0 0 0

Empty DF returned using async RDP library with multiple RICs and platform session

Hi,

When pulling prices for all index components using async RDP library part of results are returned empty, it happens only with platform session which I need for calculator I am creating, whereas with desktop session it is significantly slower but all results are returned correctly. Could you confirm why it is happening, is there any limit that I am hitting and how can I bypass this to get all instruments queried?

Sample results:

0#.FTSE first run - 37 RICs missing, rest returned correctly

No CLOSE for 35 RIC(s): ['BLND.L', 'SKG.L', 'BP.L', 'ADML.L', 'SBRY.L', 'PHNX.L', 'FLTRF.L', 'IHG.L', 'BT.L', 'MNDI.L', 'CRH.L', 'SPX.L', 'PSN.L', 'RIO.L', 'RDSb.L', 'CCH.L', 'RSA.L', 'RR.L', 'BMEB.L', 'SSE.L', 'LLOY.L', 'NWG.L', 'NXT.L', 'ITRK.L', 'BDEV.L', 'AVV.L', 'RDSa.L', 'FERG.L', 'ICP.L', 'BAES.L', 'DCC.L', 'ULVR.L', 'OCDO.L', 'TSCO.L', 'LGEN.L'].

0#.FTSE second run 15 RICs missing, rest returned correctly

No CLOSE for 15 RIC(s): ['MNDI.L', 'BATS.L', 'SPX.L', 'CCH.L', 'SSE.L', 'SMT.L', 'POLYP.L', 'GLEN.L', 'BARC.L', 'BAES.L', 'DCC.L', 'ULVR.L', 'LSEG.L', 'TSCO.L', 'LGEN.L'].

0#.SPX 139 RICs missing, rest returned correctly

No 139 RIC(s): ['AMAT.OQ', 'FCX.N', 'MAS.N', 'UAA.N', 'PVH.N', 'CBRE.N', 'CINF.OQ', 'SEE.N', 'WMB.N', 'EA.OQ', 'TMUS.OQ', 'CB.N', 'MDLZ.OQ', 'BLL.N', 'PFG.OQ', 'JPM.N', 'TTWO.OQ', 'RMD.N', 'CMG.N', 'PCAR.OQ', 'CHTR.OQ', 'PWR.N', 'COO.N', 'SNA.N', 'CMI.N', 'OXY.N', 'F.N', 'CVS.N', 'PH.N', 'GILD.OQ', 'PNW.N', 'DE.N', 'HSY.N', 'GL.N', 'SLB.N', 'AWK.N', 'FLT.N', 'KLAC.OQ', 'AME.N', 'XLNX.OQ', 'NUE.N', 'WU.N', 'D.N', 'SRE.N', 'AVGO.OQ', 'WRB.N', 'RF.N', 'AES.N', 'ANTM.N', 'GWW.N', 'K.N', 'GOOGL.OQ', 'CCI.N', 'ROP.N', 'C.N', 'ODFL.OQ', 'LEG.N', 'A.N', 'MET.N', 'WYNN.OQ', 'PSA.N', 'BK.N', 'FAST.OQ', 'CFG.N', 'NI.N', 'MO.N', 'CMCSA.OQ', 'EIX.N', 'UPS.N', 'BAC.N', 'VIAC.OQ', 'IP.N', 'MDT.N', 'INTU.OQ', 'EQIX.OQ', 'CHD.N', 'MTD.N', 'PEG.N', 'BIIB.OQ', 'CTSH.OQ', 'ALB.N', 'NCLH.N', 'WAT.N', 'MAR.OQ', 'MNST.OQ', 'FRC.N', 'BMY.N', 'MCHP.OQ', 'CAT.N', 'PG.N', 'ZTS.N', 'STE.N', 'AFL.N', 'CPB.N', 'FITB.OQ', 'XEL.OQ', 'HSIC.OQ', 'MPC.N', 'ICE.N', 'IPGP.OQ', 'SPGI.N', 'ADBE.OQ', 'ADSK.OQ', 'WRK.N', 'FOXA.OQ', 'TSN.N', 'LUV.N', 'TSCO.OQ', 'AON.N', 'AMZN.OQ', 'KR.N', 'BKNG.OQ', 'SWK.N', 'INTC.OQ', 'TRMB.OQ', 'WBA.OQ', 'DISCK.OQ', 'OMC.N', 'GLW.N', 'FTNT.OQ', 'HBI.N', 'DOW.N', 'PPL.N', 'VFC.N', 'ETN.N', 'LNC.N', 'CI.N', 'XYL.N', 'DISH.OQ', 'LUMN.N', 'NRG.N', 'LMT.N', 'PSX.N', 'FLIR.OQ', 'SCHW.N', 'DXC.N', 'SNPS.OQ', 'J.N', 'SIVB.OQ'].

Code used:

import refinitiv.dataplatform as rdp
import asyncio
import pandas as pd

# session = rdp.open_desktop_session('')

session = rdp.open_platform_session(
    '', 
    rdp.GrantPassword(
        username = '', 
        password = ''
    )
)
session.open()


default_df = rdp.get_data('0#.FTSE',
              fields = ['TR.AssetCategoryCode']
              )


async def pull_raw_data():

            df = pd.DataFrame()
            tasks = []
            responses = []
            for ric in default_df.instrument:
                task = asyncio.ensure_future(rdp.HistoricalPricing.get_summaries_async(universe = ric, start = '04/20/2020, 00:00:00', end = '04/20/2021, 00:00:00', interval = 'P1D', fields = ['TRDPRC_1']))
                tasks.append(task)
            responses = await asyncio.gather(*tasks)
            remaining_instruments = []
            for i in range(len(default_df)):
                ts = responses[i].data.df  
                if ts is not None:
                        ts.rename(columns = {'TRDPRC_1': default_df.instrument[i]}, inplace = True)
                        df = pd.concat([df, ts], axis=1)
                else:
                    remaining_instruments.append(default_df.instrument[i])          
           
            if len(remaining_instruments):
                print(f"No {str(len(remaining_instruments))} RIC(s): {str(remaining_instruments)}.")
            display(df)
await pull_raw_data()



@marcin.bunkowski, @David Sobolewski

pythonrdp-apirefinitiv-data-platformrefinitiv-data-platform-libraries
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
Accepted
13k 32 12 18

Hi @katarzyna.olenycz,

You are mixing up the call from platform and desktop session, so your posted code does not work. Either way, I have found out the reason of the failure - it is too many data requests.

Please refer to the Rate limit section for the historical pricing service in the API playground. It does not specify the actual limit, but does mention that if the rate is exceeded, then the application will receive HTTP-429.

Here is the actual message I received for one of the missing data instrument, using your code and FTSE 100 index:

HTTP/1.1 429 Too Many Requests

{
  "error": {
    "id": "47828f29-454c-4423-aa73-3059b59d6e0c",
    "code": "gw.userLimit",
    "message": "too many requests for /data/historical-pricing/v1/views/intraday-summaries/{universe} [GET]",
    "status": "Too Many Requests"
  }
}


You will have to pace out the requests to not breach this limit.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
13k 32 12 18

You can also check the error response in the failed instrument, by modifying your code like:

for i in range(len(default_df)):
    ts = responses[i].data.df  
    if ts is not None:
        ts.rename(columns = {'TRDPRC_1': default_df.instrument[i]}, inplace = True)
        df = pd.concat([df, ts], axis=1)
    else:
        display(responses[i].data.raw)
        remaining_instruments.append(default_df.instrument[i])          

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Click below to post an Idea Post Idea