What is Oracle Cloud Infrastructure Storage Gateway?
Oracle Cloud Infrastructure Storage Gateway helps you bridge your on-premises data center with Oracle Cloud Infrastructure Object Storage. Using Storage Gateway, any application that can store data on an NFS target can easily and seamlessly move data to Object Storage without having to first modify the application to uptake REST APIs. Storage Gateway exposes an NFS mount point that can be mounted to any host that supports an NFSv4 client. Storage Gateway performs the NFS to REST API translations for you. A local configurable cache enables asynchronous and optimized movement of the data to the cloud to work around bandwidth limitations or connectivity issues. Because frequently accessed data is cached locally, the cache also improves latency when reading data from the cloud. Each Storage Gateway NFS mount point maps to an Object Storage bucket. The NFS interface lets you use the traditional file system commands to interact with cloud storage.
Oracle Cloud Infrastructure Storage Gateway supports file to object transparency. This means that when the Storage Gateway uploads your data to the cloud, it preserves the original data format. The data in the cloud can be read or managed using supported Object Storage clients, such as APIs, SDKs, the CLI and Console, or other third party tools.
What are the benefits of using Storage Gateway?
Using Storage Gateway, you can easily move data generated by traditional applications that support common file system protocols like NFS to Object Storage without having to modify the application to integrate with the REST APIs. Storage Gateway opens up virtually unlimited cloud storage to traditional on-premises applications, which previously could not take advantage of the seamless scalability, durability, and cost efficiency of Object Storage.
Can I use Storage Gateway as a general purpose Network Attached Storage server?
No, Storage Gateway must not be used as a general-purpose network-attached storage. Though Storage Gateway is in many ways a virtual shared file system, its primary function is data movement to the cloud. Storage Gateway is not built to handle the transactional workload that a full-featured shared file system can handle.
Which use cases does Storage Gateway support?
Storage Gateway is an efficient tool to move data from your on-premises data center to the cloud. You can use this service for use cases like backup/archive, disaster recovery, data migration, and hybrid (on-premises and cloud) workloads. Using Storage Gateway, you can easily script the movement of cold data from on-premises NAS servers to Oracle Cloud Infrastructure Object Storage.
Which use cases does the Storage Gateway not support?
General purpose network storage - Storage Gateway is not a general purpose storage filer and must not be used as a replacement for traditional network storage appliances.
File sync and share - Though Storage Gateway is an effective data mover, it's not a replacement for file sync and share services. Evaluate Oracle services like Oracle Document Cloud service for your file sync and share needs.
Content collaboration - Storage Gateway does not support multiple instances simultaneously reading from and writing to a single Object Storage bucket. Storage Gateway is not meant to be used as a tool that enables distributed teams to collaborate on creating and managing content.
Frequently modified files – When an existing file is modified, its content is read from Object Storage if not present in cache. The modified file is later uploaded as a new version/instance of the file. Therefore, files that are frequently modified are not ideal both in terms of consuming upload/download bandwidth and capacity utilization.
How much does Storage Gateway cost?
Storage Gateway is a free service for all Oracle Cloud Infrastructure customers.
Which storage protocols does Storage Gateway support?
Storage Gateway supports the NFSv4 protocol.
Does Storage Gateway support the Windows operating environment?
No, Storage Gateway does not currently support the Windows operating environment.
Can I use Storage Gateway to move data to the cloud from third party storage arrays?
Yes, you can use Storage Gateway to replicate data from any storage, as long as the NFS client that is performing the data copy can either facilitate the replication or can use the Storage Gateway cloud sync feature.
I'm using Oracle Cloud Infrastructure Classic Object Storage and Storage Software Appliance, can I move my data to Oracle Cloud Infrastructure Object Storage and Storage Gateway?
Yes, we can facilitate this migration. Contact us about using the Storage Gateway cloud migration utility.
Do I need to use Object Storage REST APIs to use Storage Gateway?
No, you do not need to use any Object Storage APIs to use Storage Gateway. Storage Gateway performs the NFS to REST API translation for you.
How does Storage Gateway leverage Object Storage?
When you create a file system on the Storage Gateway, an Object Storage bucket with the same name is created in the Oracle Cloud Infrastructure region of your choice. Storage Gateway completely manages your data in the Object Storage bucket for you. You only need to interact with the local file system using normal file system commands.
Does Storage Gateway support compression or deduplication?
No, Storage Gateway does not support compression or deduplication.
How can I get support for Storage Gateway?
You can get support for Storage Gateway by contacting Oracle Support at https://support.oracle.com.
Can I create more than one file system on a Storage Gateway instance?
Yes, you can create multiple file systems on a single Storage Gateway instance. The hardware the host is running on dictates the maximum number of file systems that can be supported by a Storage Gateway instance. We recommend that you create no more than 5 active file systems per Storage Gateway instance. For best performance, host each file system on a dedicated Storage Gateway.
What are the Storage Gateway scalability limits?
Storage Gateway can manage a maximum of 100 million files per file system. For data sets that contain more than 100 million objects, ensure that the objects are distributed across multiple file systems.
What is the maximum size of a file that can be uploaded using Storage Gateway?
The maximum supported file size is 75% of cache space configured on Storage Gateway, but cannot exceed 10 TB. For example, if Storage Gateway is configured to have 1 TB of cache space, the maximum supported file size is 750 GB.
Can I instantiate multiple Storage Gateway instances?
There is no upper limit on the number of Storage Gateway instances that can be installed for use within an Oracle Cloud Infrastructure tenancy.
How much data can I upload to the Object Storage using Storage Gateway?
As long as the local cache is sized accurately, you can upload an unlimited amount of data to Object Storage. Note, that Storage Gateway file systems should not exceed 100 million files.
What is the longest file or directory name that is supported?
The longest supported file name can be 255 characters. The total path length can be a maximum of 1024 characters, which is the maximum supported length for Object Storage object names. For example, /directory1/directory2/directory3/directory4/filename.txt is 57 characters in length. Note that the maximum supported length for Object Storage paths is shorter than the maximum of 4096 characters supported by Linux file systems. Trying to create a file or directory for which path is longer than 1024 characters generates an NFS error.
How many directories are supported?
There is no limit to the number of directories, as long as the path length of each directory is 1024 characters or less.
How many files are supported in a single directory?
There is no limit to the number of supported files in a single directory.
Can I run Storage Gateway in an Oracle Cloud Infrastructure Compute instance?
Yes, you can run the Storage Gateway in an Oracle Cloud Infrastructure compute instance using the same general installation instructions you follow for installing Storage Gateway on-premises.
Can Storage Gateway be installed behind proxy server?
Yes, Storage Gateway can be installed behind a proxy server. To set the proxy servers, use the following commands:
- sudo ocisg configure proxy <http_proxy_server> <https_proxy server>
- sudo ocisg down
- sudo ocisg up
Can multiple Storage Gateway instances run concurrently against the same bucket?
No, multiple Storage Gateway instances cannot run against the same Object Storage Bucket. Storage Gateway does not enforce object locking or consistency between different Storage Gateway instances.
When I upload data to Object Storage using Storage Gateway, can the data stored in the Object Storage be accessed using tools other than Storage Gateway?
Yes, data uploaded to the cloud using the Storage Gateway can be managed using any supported Oracle tools like APIs, SDKs, the CLI and Console, or third party tools like Cyberduck or CloudBerry.
Does Storage Gateway ensures consistency between local storage and Object Storage?
Changes made to a Storage Gateway file system are eventually consistent with the Object Storage. Uploads are asynchronous and buffered for performance, so Storage Gateway file changes might not yet be reflected in Object Storage.
Are changes made directly to the Object Storage bucket, reflected back to the local Storage Gateway file system?
Changes made directly to the Object Storage bucket are not reflected back to the local Storage Gateway file system in real time. Click 'Refresh' in the Storage Gateway management console to ingest any data that was added or modified directly in Object Storage.
Can a Storage Gateway file system be created against a non-empty target Object Storage bucket?
Yes, a Storage Gateway file system can be created against a non-empty bucket. When this happens, Storage Gateway creates metadata for all of the objects in the bucket with default permissions and attributes. Storage Gateway also creates a local inventory of the objects in the bucket. Depending on the number of objects in the target bucket, this process might take some time to complete. The Storage Gateway management console provides a progress bar showing the current state.
Does the Storage Gateway support hard or symbolic links?
No, Storage Gateway does not support symbolic or hard links. Trying to create a link will result in an NFS error.
Does the Storage Gateway support sparse files?
Yes, Storage Gateway supports the creation of sparse files. However, the corresponding Object Storage objects will be non-sparse and filled with 0x00s.
If I delete a file system, will all of my data in the cloud be automatically deleted as well?
No, if a file system is deleted/removed, the data stored in the cloud is left intact. You can reclaim the deleted data by creating another file system backed by the same bucket.
Does deleting data from the Storage Gateway permanently delete the data from Object Storage?
Yes, deleting data from Storage Gateway permanently deletes the data from Object Storage.
How long does the Storage Gateway wait before uploading files to the Object Storage bucket?
Storage Gateway does not upload a new file or modified file until the file is closed. After a file is closed, Storage Gateway allows a 1 minute settling period before queuing the file for upload.
Depending on the number of files pending upload, it might be a while before the file is uploaded to the Object Storage bucket. You can view the upload status of files from the Storage Gateway management console.
Integrated Cloud Synchronization
What is the new Storage Gateway Cloud Sync feature?
Storage Gateway 1.2 provides an integrated cloud synchronization and replication feature that lets you backup and replicate NAS files to and from Oracle Cloud Infrastructure Object Storage buckets. This new feature replaces the independent, downloadable cloud sync utility that was available in previous Storage Gateway versions.
You can use the Storage Gateway management console or CLI to easily create, monitor, and manage sync jobs similar to other enterprise NAS backup/replication offerings. Cloud Sync runs as part of the Storage Gateway software inside the Docker instance on the host.
See Using Storage Gateway Cloud Sync documentation for details.
When would I want to use Storage Gateway Cloud Sync and for what kind of data?
Customers with a lot of on-premises files to move into the cloud can do so using the Oracle Cloud Infrastructure Storage Gateway. Cloud Sync offers a way to copy files from the local storage into the Storage Gateway, where the files are asynchronously and efficiently moved to an Object Storage bucket. Customers who store database or data backups in local NAS and want to free up the space, or have a desire to move older backups into less expensive cloud storage, will find Cloud Sync an easy way to automate this kind of data movement.
Can I use the Storage Gateway Cloud Sync to copy data into both Standard tier and Archive tier buckets?
Yes, when you create a file system, you specify which storage tier, Standard (the default) or Archive, to create the corresponding Object Storage bucket. The Standard Object Storage tier is used for storing data to which you need fast, immediate, and frequent access. The Archive Storage tier is used for storing data that is accessed infrequently and requires long retention periods. While Archive Storage is more cost effective than Object Storage for preserving cold data, you must first restore the objects before you can access them.
What kind of Cloud Sync control and status information is provided?
Each Cloud Sync copy operation provides comprehensive status to let you know the list of files selected, when they have been copied over to the Storage Gateway, and when they have been successfully uploaded to Object Storage. When a copy operation is in progress, Cloud Sync provides the start timestamp, elapsed time, estimated completion time, the total number/size of files to be synced, and the total number/size for files that have been already been synced.
How are files that are copied over the network verified to be consistent?
After the file transfer is complete, the cloud sync utility compares the checksums of the files in the source and the target locations.
What upload and download speeds can I expect from Storage Gateway?
In performance benchmark tests with 10 Gb/s link speed, we are seeing upload and download speeds similar to Storage Gateway.
What upload and download speeds can I expect from Storage Gateway?
In performance benchmark tests with 10 Gb/s link speed, we observed the following:
- Upload Speeds
- Single File Upload – 100 - 120 MB/s (using a 1.3 TB file)
- Multiple File Upload – 450 - 500 MB/s (using 4 to 8 files ranging from 10 GB to 50 GB in size)
- Download Speeds
- Single File Download – 300 - 350 MB/s (using a 1.3 TB file)
- Multiple File Download – 700 - 750 MB/s (using 3 files ranging from 50 GB to 200 GB in size)
How can I ensure my network is configured correctly to make use of the full bandwidth available?
Storage Gateway 1.2 provides a diagnostic command that you can use to test the bandwidth in your environment and ensure that you are getting the expected upload and download speeds.
See Testing Network Bandwidth documentation for more details.
What are the memory and cache requirements to get the maximum performance?
Follow the documented best practices to get the maximum performance benefits from Storage Gateway.
Are there any tunable parameters that I can adjust to ensure that I am getting the maximum performance possible from Storage Gateway?
Storage Gateway 1.2 is tuned for maximum upload and download performance by default. No additional tuning is needed.
What other factors can impact Storage Gateway performance?
In addition to having sufficient memory and file system cache space, we recommend using SSDs in a Storage Gateway environment. Storage Gateway needs to read from the cache to upload to Oracle Cloud Infrastructure. If the backend media is slow, NFS ingestion rate and thus the overall upload performance is negatively affected.
What upload speeds are possible if I use FastConnect in conjunction with Storage Gateway?
Using FastConnect in conjunction with Storage Gateway makes optimal use of full link speed. We have customers using FastConnect with 10Gb/s link speed and have seen each gateway achieve 400 - 450 MB/s uploads to Oracle Cloud Infrastructure.
See FastConnect documentation to learn more.