What is Oracle Cloud Infrastructure Data Transfer?
Data Transfer is an offline data migration service that lets you securely move petabyte-scale datasets from your datacenter to Object Storage or Archive Storage on Oracle Cloud Infrastructure. Using the public internet to move data to the cloud is not always feasible due to high network costs, unreliable network connectivity, long transfer times, and security concerns. The Data Transfer service overcomes these challenges and can significantly reduce the time that it takes to migrate data to the cloud.
When should I use Data Transfer?
You should use Data Transfer when uploading data using public internet connectivity is not feasible. This lack of feasibility may stem from the upload dataset being too large, the internet connection being too slow, or an inability to dedicate the required internet bandwidth for the extended time that it takes to upload data. Data Transfer is also a great alternative when using Oracle FastConnect is not an option, either because your network infrastructure cannot support it or when you cannot justify the FastConnect investment because you do not expect to transfer data frequently.
You can comfortably use Data Transfer to upload hundreds of TB of data to the cloud. Our recommendation is that you consider using Data Transfer if uploading data over the public internet takes longer than 1-2 weeks. Looking at the following table, if you have a 100-Mb internet connection that you can solely dedicate to transferring 100 TB of data, it takes more than 100 days to complete data transfer over that connection. You can complete the same transfer using Data Transfer in about a week, including shipping.
|Approximate Data Upload Time*|
|Dataset Size||10 Mbps||100 Mbps||1 Gbps||10 Gbps||Data Transfer Service|
|10 TB||92 Days||9 Days||22 Hours||2 Hours||1 Week|
|100 TB||1,018 Days||101 Days||10 Days||24 Hours||1 Week|
|500 TB||5,092 Days||509 Days||50 Days||5 Days||1 Week|
|1 PB||10,185 Days||1,018 Days||101 Days||10 Days||2 Weeks|
*The data upload times depicted in the table are approximate and must not be construed to be an SLA for the Data Transfer service. The actual time it takes to upload data varies.
When should I use Oracle FastConnect instead of Data Transfer?
Oracle FastConnect is a network-connectivity alternative to using the public internet for connecting your on-premises datacenter or network with Oracle Cloud Infrastructure. Typically, you would use Oracle FastConnect when you require a dedicated, reliable, and consistent networking experience as compared to public internet-based connections.
FastConnect is a great alternative to Data Transfer. We encourage the use of FastConnect to transfer the data whenever possible. This is especially true if you need to transfer large quantities of data to Oracle on an ongoing basis. However, if investing in FastConnect is not feasible or you don't expect to frequently upload data from your datacenter, then Data Transfer is a great alternative to migrate your data to the cloud.
Two solutions: Data Transfer Appliance and Data Transfer Disk
The Data Transfer Appliance service uses a high-capacity storage appliance that you loan from Oracle for a short duration of time (30 days) to facilitate data migration to the cloud. The transfer appliance is especially useful when you need to migrate a large amount of data, up to 150 TB per appliance. Currently, the Data Transfer Appliance service is only available to customers in the United States.
The Data Transfer Disk service requires that you purchase your own hard drives to migrate data to the cloud. These hard drives are shipped back to you after the data is transferred successfully to your Object Storage or Archive Storage tenancy. The disk option is practical if you transfer only a relatively small set of data, or if you need more time than 30 days to copy your data for transfer.
This FAQ page covers Data Transfer Disk. For more information about Data Transfer Appliance, see the Data Transfer Appliance FAQ webpage.
Data Transfer Disk
The following information describes the details and frequently-asked questions for using Data Transfer Disk.
What are the core components of Data Transfer Disk?
Transfer Job: A transfer job is a logical representation of the data set that is uploaded to Oracle Cloud Infrastructure. When you create a transfer job, you signal the intent to move data to Oracle Cloud Infrastructure Object Storage. All data associated with a transfer job can only be moved into a designated Object Storage bucket of your choice. A transfer job has at least one transfer package and one transfer disk attached to it.
Transfer Disk: A transfer disk is a hard disk drive that you own, which has been specially prepared using the data transfer utility. The data that you intend to migrate needs to be copied over to the transfer disk, before it can be uploaded to Oracle Cloud Infrastructure Object Storage.
Transfer Package: A transfer package is a logical representation of a package that you ship to Oracle. Each transfer package can include up to ten transfer disks.
How does Data Transfer Disk work?
First, create a transfer job. A transfer job represents a dataset that you want to import to Object Storage. Next, you must prepare one or more transfer disks. Each transfer disk represents a hard drive disk (HDD) that you intend to use to transfer the data. Each HDD represents a portion of your data to be copied into the target Object Storage bucket. For each transfer disk you create, you must copy your data to a compatible HDD. Finally, you create a transfer package and attach the transfer disks to it. A transfer package represents a single package that you send to Oracle, which can be tracked using a unique carrier tracking ID.
Once Oracle receives the transfer package at the Oracle data transfer site, an Oracle datacenter operator copies the data from the transfer disks to the designated Object Storage bucket. You are able to track the status of each transfer package and transfer disk using the Data Transfer Utility or the Oracle Cloud Infrastructure Console. Once the data transfer is complete, the transfer log file is copied over to the bucket where your data was uploaded. After you verify that data has been uploaded as expected, close the transfer job. Oracle then securely deletes your data off the disks and returns the transfer disks to the return address you provided.
How do I get started with Data Transfer Disk?
You can get started on Oracle Cloud Infrastructure Data Transfer Disk using the Console or Data Transfer Utility. However, you need to make sure that you first sign up for Oracle Cloud Infrastructure in one of the following ways:
- Contact your Oracle sales representative. Your Oracle sales representative can tell you about the pricing options available to your company. Your sales representative will collect some information from you and initiate the registration process.
- Go to the Oracle Store and sign up for Oracle Cloud Infrastructure.
Which Oracle Cloud Services can I use with Data Transfer Disk?
You can move data into any Oracle Cloud Infrastructure storage service using Data Transfer Disk. At the end of the data transfer process, the imported data is available in Oracle Object Storage. From there, you can copy the data to the other services as needed, using the Oracle Cloud Infrastructure tooling. For RMAN backups, deep integration exists to help move data from Object Storage into Database instances.
How much data can I import using Data Transfer Disk?
You can use Data Transfer Disk to import up to 100 TB of data in a single transfer package. If you need to import more than 100 TB of data, you need to break the data into multiple transfer packages. Review the default Data Transfer Disk service limits to determine the number of transfer jobs, transfer packages, and transfer devices you can create in your tenancy. If you want to request a limit increase, reach out to Oracle support.
What type of disks can I use to transfer data?
You can transfer data using the following disk types:
- USB 2.0/3.0 external HDDs
- SATA II/III 2.5"/3.5" internal HDDs
Each HDD has one partitioned filesystem with your data on it.
You are responsible for purchasing the required number of hard drives, copying the data to the HDDs following the provided instructions, and shipping the disks to the Oracle data transfer site. Once the data is transferred successfully to Oracle Cloud Infrastructure Object Storage, the hard drives are shipped back to you in the same encrypted state they were received in.
What are the software requirements for using the Data Transfer Utility?
- Linux operating system with the ability to create an EXT4 filesystem
- cryptsetup package 1.0.3 or greater
- hdparm 9.0 or greater
- Java 1.7 or greater
Can I loan hard drives from Oracle instead of having to purchase the drives specifically for the purpose of data transfer?
No, currently Oracle does not support loaning HDDs due to various issues related to security and compliance. However, Oracle offers a Data Transfer Appliance that can be loaned. For more information see the Data Transfer Appliance FAQ webpage.
Can I create unlimited number of transfer jobs, transfer disks, and transfer packages?
Can I use Data Transfer Disk in all Oracle cloud regions?
The Data Transfer Disk service is available in us-phoenix-1, us-ashburn-1 and eu-frankfurt-1 regions at the moment.
How long does it take to load my data?
The time it takes to load data into your account once Oracle physically receives it, depends on several factors including: type of hard drive used, average file size, and total volume of data being imported. Transfer devices are processed on a first-come, first-served basis. Smaller average file size and large volumes of data take longer to import. Currently, Oracle does not make any SLA guarantees related to data upload times.
How can I validate that my transfer disk is configured correctly before sending it?
You can use the Data Transfer Utility to validate the configuration of your transfer devices before sending it to Oracle. This helps catch common mistakes and saves time. Refer to product documentation for details.
How do I know that my data was copied successfully?
You can check the status of each transfer package and transfer disk using the Data Transfer Utility or Console. After a transfer disk has been processed, a transfer log is posted to your designated Object Storage bucket. Files that were successfully imported include the flattened object name and the calculated MD5 checksum. You must review the transfer log before deleting original copies of the data.
Can I ship transfer devices from international location to a Oracle Cloud Infrastructure region where Data Transfer Disk is supported?
Yes, you can ship transfer devices from international locations to Oracle Cloud Infrastructure regions where Data Transfer Disk is supported.
What happens to my hard drive after the data has been imported?
After the data from each transfer device has been copied to your target bucket, the data on the hard drives will be securely deleted. It is then shipped back as a single package to the return address specified when creating the transfer package.
What is the file size limit?
Files up to 10 TB can be uploaded. Files larger than 10 TB are skipped in the import process.
What shipping vendors can I use to ship my data?
Customers can use one of the following couriers to send packages: FedEx, UPS, or DHL.
Can my data be loaded into more than one Object Storage bucket?
Datasets attached to a specific transfer job can only be moved into one designated bucket. If you need to upload your data to multiple Object Storage buckets, you must create multiple transfer jobs.
Do I need to keep a backup of my data during import?
Yes. You must keep a backup of your data until you have confirmed that your data has been successfully copied to the target bucket.
What are the filesystem requirements for HDD import?
To import data using HDDs, you must format the filesystem as an EXT4 filesystem.
How are my files from the local file system mapped to objects?
Files are uploaded to the target bucket as objects. The name of the object is the flattened path of the file.
What happens if my bucket has an object with the same name as a file being imported?
If a file is being imported into a bucket and the bucket already has an object of the same name, the file fails to upload and the existing object remains intact. The failed upload status of the file is logged in the upload summary report.
Can I preserve the source filesystem metadata when importing data into Oracle Object Storage?
No, filesystem metadata is not preserved when being imported to the target bucket. The object name becomes the flattened path to the file. The timestamp is the time the object was created.
Does Oracle maintain a chain of custody for my package?
Yes. Oracle rigorously maintains the state change at the transfer job, transfer package, and transfer disk level. You can review the status using the Console or the Data Transfer Utility.
Will the Data Transfer Service transfer the symbolic links that exist in my source filesystem?
If you want to maintain the source-side symbolic links, tar up the source directories and copy data over as a single object.
Can I cancel a transfer job, transfer package, or transfer disk from being imported to Oracle Object Storage?
Yes, you can use the Console or the Data Transfer Utility to delete a transfer job and cancel a transfer package or a transfer disk. A transfer job cannot be deleted after the transfer job status changes to Active. The transfer package and transfer disk can be canceled, up to the point where the status changes to Processing. If the transfer package or transfer device is canceled, it is shipped back to you without uploading the data to the Object Storage.
Can a transfer job have more than one transfer package attached to it?
Yes, a transfer job can have multiple transfer packages attached to it. Each transfer package however needs to be tracked by a unique carrier tracking ID. You can review the default data transfer service limits here.
Can I use two different shipping addresses to ship from and receive the transfer packages?
Yes, you can use multiple shipping addresses to ship from and receive your transfer package. You can even ship transfer packages from multiple shipping locations, using a single transfer job.
Can I upload data to a third party's Object Storage bucket?
Yes, you can upload data to a third party's bucket as long as you can supply the necessary Identity and Access Management (IAM) credentials to establish access to that bucket. The third party that owns the bucket is responsible for all the associated Object Storage costs.
How is the integrity of the data being uploaded maintained?
When you copy the data to the transfer disk, a manifest file captures MD5 checksums of all the files. When data is uploaded to your bucket, we match the MD5 checksum from the manifest with the MD5 checksum of the uploaded data to ensure that the integrity of the data has not been compromised during the upload process.
How do I find the address to ship my transfer package?
You can query for the region-specific shipping address using the Console or the Data Transfer utility.
My transfer package is lost. What do I do?
If you have reasons to believe that your transfer package is lost, contact your shipping carrier to determine the status of the package.
What happens if my transfer package or transfer disks are damaged during transit?
If Oracle determines the transfer package or a transfer disk is damaged, we skip the upload process for that specific package or disk and then ship the HDDs back to you normally. If a transfer disk could not be processed, it is set to the 'Error' state. If a file could not be uploaded to your designated bucket, it is noted in the upload summary.
Oracle makes no representation and assumes no liability towards the safety of the data transfer medium. You must maintain a backup of your data until you can verify that the data has been uploaded to your Object Storage bucket as expected.
How is my data protected while in transit?
Your data is encrypted using an AES-256 cipher while in transit to Oracle. The encryption key is not sent with the disk, it is provided to Oracle through the internet. Once the disks are received by Oracle, the data is decrypted before being imported to your target bucket. During upload, your data is encrypted using SSL/TLS.
Will Oracle wipe my device?
Yes. Oracle deletes the data off the disk and then returns your transfer disks back to you.
Who handles my package and devices?
Once Oracle receives your package, your disks are only handled by Oracle employees and contractors.
How much does Data Transfer Disk cost?
The Oracle Data Transfer Disk service is free. You are only charged for Object Storage usage once the data is successfully transferred to your Object Storage bucket.
Additional out-of-pocket costs include the cost to purchase HDDs and the cost to ship data to and from the Oracle transfer site.
Who pays for shipping when using Data Transfer Disk?
You are responsible for paying shipping fees on both inbound and outbound shipments. When shipping to Oracle the customer is responsible for paying shipping and international customs fees (if required). When returning HDDs to customers, Oracle requires that you provide a valid return shipping label and that you pay international customs fees (if required).