Error message Forbidden when using webread in Matlab

I try to download data from this site:

https://fragmentation.fidessa.com/fragulator/

I have ISIN numbers that I want to download data for. Here I will use the example BMG0451H1170.

When I do it manually it looks like I can do this in two steps:

Step 1:

Go to page

https://data.fidessa.com/FragPro2/StockSearch?term=BMG0451H1170

Get the code 0RBA.L

Step 2:

Go to page

https://data.fidessa.com/FragPro2/fragBreakdownHistory?fim=0RBA.L&start=20210630&end=20220629&global=Y&format=json

Get the data

However, I encounter the following problem when I try to use webread:

When I use

url='https://data.fidessa.com/FragPro2/StockSearch?term=BMG0451H1170'

options = weboptions('Timeout', 5000, 'HeaderFields',{'Content-Type' 'application/json'},'UserAgent','');

searchResult=webread(url,options);

I get the following error message:

The server returned the status 403 with message "Forbidden" in response to the request to URL

The only information I’ve found about this suggested that I’d play around with the UserAgent. I did, but without success. I’ve also tried without the json option.

I get the same error message in step 2. I should use a POST request there, which may be part of the problem. However, in the first step a GET request seems to be enough.

I know I should only ask one question, so I would primarily be very grateful if somebody could help me get past the 403 error. If you have additional information on how I should use the POST request in step 2, I would also appreciate that very much.

Thank you for your help.

[Geoff Hayes]

@Marcus - are you using any login information for this site? i.e. have you created an account which seems to be an option for the first link.

[Marcus]

@Geoff Hayes, no, I haven't. I can load the side manually though, without a login, so I don't think it's necessary.

The link to register seems to lead to the wrong page, and the Contact Us returns Page not found. Possibly they're updating the site. I'm trying to get in contact with them.

I've checked the robots.txt file and there is no disallow for /Frag2Pro/. According to their FAQ, registering is free, so I don't think the error is intentional on their part.

[Marcus]

Maybe the full error message can give som guidance:

Error using matlab.internal.webservices.HTTPConnector/copyContentToByteArray (line 373)

The server returned the status 403 with message "Forbidden" in response to the request to URL

https://data.fidessa.com/FragPro2/StockSearch?term=BMG0451H1170.

Error in readContentFromWebService (line 46)

byteArray = copyContentToByteArray(connection);

Error in webread (line 125)

[varargout{1:nargout}] = readContentFromWebService(connection, options);

[Geoff Hayes]

@Marcus - a 403 error (to me anyway) means that you are not providing the proper (authroization) credentials along with your request. Or perhaps like you said, there are problems with the site.

See Troubleshooting RESTful API Errors for details.

[Marcus]

Thanks! It was the cookie that was missing.

Answer 1

[Marcus]

I answer my own question. The only thing missing was a cookie:

options = weboptions('Timeout', 5000, 'ContentType','json',...

'KeyName', 'cookie', 'KeyValue', ['JSESSIONID=', sessionID]);

[Robert]

Hello, I believe I have a similar problem. I'm trying to read the page

url='https://www.idealista.com/de/venta-viviendas/valencia-valencia/pagina-2.htm';

C=webread(url);

I also get the error "The server returned the status 403 with message "Forbidden" in

response to the request to URL".

In your solution, I don't understand the webopitions KeyName and following, especially what is the value of sessionID?

[Eric Machorro]

I have the exact same question/issue.

[Marcus]

This was some time ago, but I think I got the sessionID by first visiting manually, and then getting the sessionID by inspecting that page. But again, it was some time ago, and I don't quite remember.

It was not a beautiful solution. :-)

[Lily Yan]

Thanks Marcus for the code snippet. I was able to resolve my problem using the session ID fetched from my browser... To answer the question(s) of how to get session ID (also a note for myself), take Chrome or Edge for example, visit the url manually then press Ctrl-Shift-J to open the dev-tool, then Application > Storage > Session storage and inspect the key-value pairs:

For the session in the screenshot the weboption would look like:

options = weboptions('KeyName', 'cwvSessionId', 'KeyValue', 'f2800315-d7xxxxxxxxxx');