Upload Files to ScienceBase

Almost any file type may be uploaded to ScienceBase. Multiple files may be uploaded to a single item, and these files may be uploaded individually or bundled together in a zipped file. The current file size limit for uploads and downloads in ScienceBase is about 30 GB (dependent on users’ local internet connections). The limit for number of individual files per item is 100 (note: please upload fewer files so that users can more easily view the list).

To upload files and enter descriptive information, the user must be logged in and have edit permission to the ScienceBase item. ScienceBase offers one edit form with multiple tabs for entering descriptive information and uploading files. Creating a new item will direct the user to a blank edit form. To create a new item, click the "Add Item" link in the black title bar or the "Add" drop-down menu in the upper right corner of the folder view.

• The Add Item link in the top banner:

 

• The Add dropdown menu in the upper right corner of the page:

Users can access the edit form of an existing item by clicking on the "Manage" dropdown menu and selecting "Edit."

Once in the edit form, the user selects the "Files" tab and clicks "Add files…" After a file has been uploaded, the user can enter a title into the text box on the right side of the uploaded file. The title will appear below the file name on the item view. While it is useful to provide this title for a file, it is also recommended that users provide descriptive and meaningful file names prior to uploading their files. This practice is helpful to downstream users who will be downloading the files. 

 

Uploading FGDC-compliant XML metadata files

All USGS datasets should have associated metadata that adhere to FGDC endorsed standards – either Content Standard for Digital Geospatial Metadata (CSDGM) or the ISO metadata standard. Metadata files can be uploaded and stored as resources in ScienceBase items.

Please note: the ScienceBase edit form does not contain all the fields that are required by FGDC endorsed metadata standards. Filling out the edit form is not an alternative to uploading FGDC compliant metadata.

For more information about metadata standards and creation tools, visit the USGS Data Management website.

ScienceBase has the capability to recognize and parse FGDC endorsed Content Standard for Digital Geospatial Metadata (CSDGM) and ISO metadata files that are in XML format. This means that information from these files can be used to populate ScienceBase fields. If a CSDGM or ISO XML file is uploaded, ScienceBase will recognize the format and automatically bring up the following menu:

Select "Yes" to automatically populate many of the fields in the ScienceBase edit form. If you have already entered content in the edit form, it will be overwritten. Select "No" if you would like to upload the metadata file without populating any of the fields.

ScienceBase will apply a stylesheet to CSDGM and ISO metadata records so that they can be viewed in a human-readable format.

Uploading Large Files

To upload files larger than 1GB, please use the ScienceBase Cloud Upload / File Manager. This tool allows users to upload files directly to cloud storage in the USGS Cloud Hosting Solutions (CHS) environment. This upload path can be accessed by logging in and scrolling down to the "Item Actions" section of the ScienceBase item view and selecting "Access File Manager."  Users will be prompted through another login step to use the file manager from an item.  

Once in File Manager, users can click the "Add" dropdown menu and choose to upload files from their local computer or from a readable S3 bucket location. While performance will depend on a user's local internet connection, files up to ~30 GB in size can be successfully transferred. Once a file has been uploaded to the Cloud, it will be available for download from the ScienceBase item. Users who would like to provide access to their cloud optimized files should refer to the Advanced Usage for Cloud Optimized Files section below. 

Note that currently no automated scraping or processing will occur for files that go through the cloud file manager (e.g., no automatic metadata ingest into item, no map service generation, etc.). Users needing those features for given files should continue to use the traditional file upload in the "Manage Item">"Edit Item">"Files" interface.  

Currently, cloud storage and access costs are supported by the Science Analytics and Synthesis Program (SAS) for public data releases in ScienceBase. However, scientists working with very large datasets or large collections of data who are interested in using these features should reach out to the ScienceBase team for additional details (sciencebase@usgs.gov).

Advanced Usage for Cloud Optimized Files 

Users who have uploaded cloud optimized files may need to take an additional step to make the files available to cloud workflows. Click the "Actions Menu" dropdown arrow next to the cloud optimized file and select "Publish." This action will add a copy of the file to the public ScienceBase S3 bucket and will add the public S3 URI to the item JSON, which then can be consumed by the public.

Uploading Spatial Data

Certain geospatial file formats (see list below) are recognized by ScienceBase and can be displayed in preview maps and used to generate web services. 

Uploaded spatial zip files must be unzipped for ScienceBase to recognize the format. Spatial files uploaded in zipped format will not display in preview maps or generate web services; however, they will remain available for download. 

File formats that can be processed by ScienceBase to enable interactive online viewing include the following:

• Shapefiles (.shp) -- shapefiles can be displayed without symbology in the preview map. Keyhole Markup Language, Web Map Services, and Web Feature Services are created.
• GeoTIFFs (.tif) -- GeoTIFFs can be viewed without symbology in the preview map. Keyhole Markup Language, Web Map Services, and Web Coverage Services are created.
• ESRI Service Definition files (.sd) -- Service Definition files allow ScienceBase to display shapefile and GeoTIFF symbology in the preview map. Keyhole Markup Language, Web Map Services, and ArcGIS Mapping Services are created.

When one of these file formats is uploaded, ScienceBase will recognize the format and bring up a popup window, asking if extensions should be created. Select "Create Extensions" to display the file in the preview map and generate web services for the data. 

For more information on creating geospatial services, please see the ScienceBase Geospatial Services page.

The current file size limit for file uploads and downloads in ScienceBase is 10GB; however, there are additional limits if files are used to create web services. Files must be uploaded through the edit form (not the Large File Uploader tool) in order to create web services. 

• Shapefiles -- ScienceBase can process shapefiles up to 500MB.
• GeoTIFFs -- ScienceBase can process GeoTIFFs with overviews (internal or external) up to 1GB. A GeoTIFF without an overview must be less than 250MB. 
• ESRI Service Definition files -- Single file uploads should not exceed 1-2GB. ScienceBase can process uploaded .sd files if the unzipped content does not exceed 4GB. 

Downloading Files

To download files from ScienceBase to a local system, visit the item view of the record of interest. If an item has files attached to it, they will be listed in the Attached Files section which appears toward the bottom of a record just below Contacts. Files may be downloaded to a local system individually or together as one zipped file. Click on the filename to download individual files or click the "download all" link, which is available above the list.

If a ScienceBase item has child items with attachments, all of the nested attachments can be downloaded together as one zipped file. To do this, you must be logged into ScienceBase. Select the dropdown menu next to the "Child Items" label and click "Download Nested Attached Files."

A window will pop up describing the size and number of files that will be downloaded. Select "Download" to proceed.

 

Downloading Large Files

For downloads of files exceeding 1GB, ScienceBase employs a temporary Amazon S3 download location, from which a user can acquire the file. There is an option to select "Notify by email when download is available." Follow the link in your email and select "Download File" to initiate the download.

Screenshot of large file downloader with a file transfer in progress: