how to download data from website?
83 views (last 30 days)
Show older comments
Dear all,
I am trying to download data from the following website
https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/
my problem is I can not get the files, meaning only the html been located to my computer. (below what I used to locat the link into my machine)
url='https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
filename='A2019.nc'
outfilename=websave(filename,url)
what I need is getting the files separtly and read them.
Thanks for the help.
2 Comments
Juan
on 27 May 2023
To download data from a website, follow these steps:
Identify the data you want to download on the website.
Right-click on the data or the download link.
Select the "Save link as" or "Save target as" option from the context menu.
Choose the destination folder on your computer where you want to save the downloaded data.
Click "Save" to initiate the download.
Wait for the download to complete. The time taken will depend on the size of the data and your internet connection speed.
Once the download is finished, you can access the downloaded data from the specified destination folder on your computer.
It's important to note that downloading data from websites should be done in compliance with applicable laws, website terms of service, and copyright restrictions. Ensure that you have the necessary rights and permissions to download and use the data obtained from websites.
SASSA
on 15 Aug 2024
repository.kaust.edu.sa/bitstream/handle/10754/673855/PhD%20Dissertation-Final_Lina%20Eyouni.pdf?sequence=4
The website you linked provides access to oceanographic data, but directly downloading individual files through code might be tricky. Here's a breakdown of what you're encountering and alternative approaches:
Understanding the Download Challenge:
- The website likely uses a different download method than websave expects. It might require authentication or interact with the server differently.
Alternative Approaches:
- Manual Download:
- Visit the website: https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/ 1. repository.kaust.edu.sa repository.kaust.edu.sa
- You'll see a list of folders representing dates in 2019. Click on the specific date folder containing the data you need (e.g., 2019/01/01).
- Inside the folder, you'll find individual files. Locate the desired file (e.g., AQUA_MODIS.20190101.L3m.DAY.SST4.sst4.4km.nc).
- Right-click on the file and choose "Save Link As" or a similar option to download it to your computer.
- Command-line tools (for advanced users):
- Tools like wget can be used to download files from websites. However, it might require additional configuration for NASA's specific setup. Refer to the wget documentation and NASA's download instructions (if available) for this approach.
Additional Tips:
- NASA's OceanColor website provides information on download methods: SASSA Status
- Consider using Python libraries specifically designed for downloading scientific data like earthpy or pydap. These libraries might offer better compatibility with NASA's data access methods.
Accepted Answer
Akira Agata
on 26 Mar 2019
How about the following?
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/';
str = webread(url);
links = regexp(str,'https://([\w-]+\.)+[\w-]+(/[\w- ./?%&=]*)?(\.nc)','match')';
data = cell(size(links));
for kk = 1:numel(links)
data{kk} = webread(links{kk});
end
By running this, all the .nc files are downloaded and stored in the cell array data.
12 Comments
Soo Mee Kim
on 11 Jun 2020
Hi, I have the same problem to to download nc files website.
When I tried the following code, both codes gave html file.
Please let me know how to download nc file from the website.
[my own]
fname = 'A2019152.L3m_DAY_PAR_par_4km.nc';
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
websave(fname, downloadURL, options);
[From Akira Agata's answer]
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
data = webread(downloadURL, options);
More Answers (7)
Tracy
on 1 Aug 2023
It looks like you are using MATLAB's websave function to download the file from the URL. However, the websave function is primarily used to download a file and save it to a specified location. It seems like you are trying to download multiple files, and it won't work as you expect because you are providing a URL of a directory, not a direct link to a specific file.
To download multiple files from the website, you will need to loop through the links in the directory and download each file individually. Additionally, you can use Python with the requests library to achieve this task more easily. Below is a Python script that demonstrates how to download multiple files from the website using the requests library:
python
import requests
import os
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
output_directory = './downloaded_files/'
# Create the output directory if it doesn't exist
os.makedirs(output_directory, exist_ok=True)
response = requests.get(url)
if response.status_code == 200:
# Parse the HTML content to find links to files
file_links = []
lines = response.text.split('\n')
for line in lines:
if '<a href="' in line:
start_index = line.find('<a href="') + len('<a href="')
end_index = line.find('">', start_index)
file_link = line[start_index:end_index]
if file_link.endswith('.nc'): # Only consider links that point to .nc files
file_links.append(file_link)
# Download each file and save it to the output directory
for file_link in file_links:
file_url = url + file_link
out_file_path = os.path.join(output_directory, file_link)
response = requests.get(file_url)
if response.status_code == 200:
with open(out_file_path, 'wb') as f:
f.write(response.content)
print(f"Downloaded: {file_link}")
else:
print(f"Failed to download: {file_link}")
else:
print("Failed to fetch the URL.")
print("All files downloaded successfully.")
0 Comments
Hitesh
on 3 Oct 2023
import requests
# Define the URL of the file you want to download
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/A2019.nc'
# Specify the local filename where you want to save the downloaded file
filename = 'A2019.nc'
# Send an HTTP GET request to the URL
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
# Open a local file with write-binary ('wb') mode and save the content of the response
with open(filename, 'wb') as file:
file.write(response.content)
print(f"File '{filename}' downloaded successfully.")
else:
print(f"Failed to download file. Status code: {response.status_code}")
- Coding a Web Scraper with Python: You can create a web scraper using Python, a versatile programming language. Python offers libraries like BeautifulSoup and Scrapy that make web scraping easier.
- Utilizing a Data Service: Another option is to use a data service or API provided by the website, if available. This method allows you to access structured data without the need for web scraping.
- Leveraging Excel for Data Extraction: Microsoft Excel can also be used for data extraction. You can import data from web pages into Excel and then manipulate and analyze it as needed.
- Web Scraping Tools: There are various web scraping tools and software applications designed specifically for data extraction from websites. These tools often provide a user-friendly interface for collecting data without extensive coding.
0 Comments
ABDUL
on 7 Oct 2024
Great insights on downloading data from websites! It’s fascinating how tools like MATLAB can simplify the process. Just as many of us need to navigate the SASSA appeal process to secure our benefits, having efficient methods to gather data online can be equally crucial in various projects. Thanks for sharing these tips!
0 Comments
ABDUL
on 7 Oct 2024
This post is super helpful for anyone looking to download data from websites! The techniques you’ve outlined are great. It reminds me of how important it is to stay organized, much like when we’re going through a having the right data can make all the difference. Thanks for sharing these valuable insights! For more
0 Comments
Andrietta
on 7 Oct 2024
"If you’re trying to download data from a website, web scraping techniques can help! There are some great resources for using MATLAB’s web functions to automate data retrieval. By the way, I was also searching for 'sassa payment dates 2024,' and realized combining this with web scraping could make tracking updates easier."
0 Comments
Andrietta
on 7 Oct 2024
Edited: Andrietta
on 7 Oct 2024
"Using MATLAB’s webread or websave functions is a great way to download data from websites. It really helped me out when I was looking to pull info related to sassa appeal for an automation project. Anyone working on something similar might find these functions useful!"
0 Comments
See Also
Categories
Find more on Downloads in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!