I need to transfer files between a network of servers all over the world. I have looked into using cURL and ftp_fput() but am having little luck. I can get a transfer going from my local to the server or visa versa but not from server to server.
Basically im trying to replicate Akamai on a very small scale. This is the scenario. I am working from home. I search for a file on Web Server 1 in Oregon. The file doesn't exist. I search my other servers and find that it is located on Server 2 in Germany. I then need to COPY the file from Web Server 2 to Web Server 1. That's it.
I have it all in place except getting the server-to-server transfer to work. Who can help? Thanks much.
One of the nice things about it is you can have multiple channels running at once.
If the files are small (because of potential memory limits) file_get_contents() works also.
In addition, fopen() & [fsockopen() {much complicated and a lot of effort }] will work.
I'd assume you'd need to setup some type of cron job running on the servers with a
list of files to be replicated, but without knowing more of your needs, it's hard to give
advice.
Also, you can look into rsync for server to server updating for the servers you have control over.
If you're trying to replicate actual web pages, then that's a different story and a lot
more complicated. You'd have to parse the files and initiate other "file get" operations
based on links discovered in the files - just like a browser does.
As with most things, there are a whole slew of ways to get information over the web with PHP.
And finally, if this is something you actually want to build yourself, you'll need to write
a client/server/switchover messaging system if you can't find something to just do it;
which I'm sure you can.
Google: "server to server file transfer" and a lot of links will come up.
CAVEAT: There are a number of sites purporting to offer PHP and other open source
SST apps, but some of the sites are phishing with trojans in waiting.
And finally, if you are trying to do this yourself, there's always the ever present need
for: "Error detection, recovery and mitigation." Just what you wanted hear, eh?
Thanks for all your input. Let me explain a little more about what i have done. First of, each of the files I'm transferring are over 1Gb so file_get_contents() wont work. Files are not transferred until requested by the user. I would prefer to do it all in CURL, but as i said im having issues.
I have looked into SST scripts but none seems to satisfy.
My thinking is that cURL cannot directly transfer a file server to server (please correct if wrong). I thought that the way to make it work would be to call a file on Web Server 2 using cURL and add some parameters (file name to transfer) to the call so that when ran it would using cURL transfer the file to Web Server 1. Sound good? Problem, Web Server 2 doesn't have a URL that will resolve to a specific file on the server. Therefor again only cURL via FTP login can be used (as i see it).
So do further define my question, with cURL, is there a way to transfer files between servers directly using a curl statement like
but one that works more like below with a location to get and set from/to. I know the following code is incorrect, i just would like to know how to get it right.
Do have any control over any of the processes on Web Server 2?
From the sound of it, you don't, but correct me if I'm wrong.
Yes, the syntax you're showing is wrong.
You have to specify the file of interest and/or send commands (set options) to
communicate with the server on the other end. You then do something with
the file bytes received on your (Web Server 1) end (i.e. manage storage).
I know I could get this to work, but I've haven't had to do this at this level with cURL.
I had to solve a similar problem (file size), but I had control of all of the servers.
It's going to take awhile to get it to work.
I'll play with it in my spare time, but I'm sure you'll get it before that.
Maybe we'll get lucky and someone who's already got a solution will pipe in.
Thanks for everyones help. I am posting my full function for those who may find it useful. To recap, this function "try's" to simulate Akamai's service.
1. GeoLoc user.
2. GeoLoc all servers
3. Calculate distance between user and each server.
4. Sort by distance.
5. Check to see if file is at closest server. If so download.
6. If file not at closest server, copy file from mother server to server missing file. Then notify user via email with download link to closest server.
Code:
function sst() {
global $_GET, $_SERVER;
/* start geo loc */
// http://www.imaginerc.com/software/GeoCalc/
require_once('standards/classes/geoplugin.class.php');
$geoplugin = new geoPlugin();
// http://www.phpclasses.org/browse/file/25516.html
include_once("standards/classes/GeoCalc.class.php");
$oGC = new GeoCalc();
/* set server ip's and FTP logins */
$servers = array($_SERVER['REMOTE_ADDR'],'xxx.xx.xx.xxx','xxx.xx.xx.xxx');
$user = array('','usr1','usr2');
$pass = array('','pass1','pass2',);
/* get user location */
$geo[0]['ip'] = $servers[0];
$geoplugin->locate($geo[0]['ip']);
$geo[0]['lat'] = $geoplugin->latitude;
$geo[0]['lon'] = $geoplugin->longitude;
$geo[0]['city'] = $geoplugin->city;
$geo[0]['countryName'] = $geoplugin->countryName;
/* get servers data */
for ($i = 1; $i < count($servers); $i++) {
$geo[$i]['ip'] = $servers[$i];
$geo[$i]['user'] = $user[$i];
$geo[$i]['pass'] = $pass[$i];
$geoplugin->locate($geo[$i]['ip']);
$geo[$i]['lat'] = $geoplugin->latitude;
$geo[$i]['lon'] = $geoplugin->longitude;
$geo[$i]['city'] = $geoplugin->city;
$geo[$i]['countryName'] = $geoplugin->countryName;
$geo[$i]['dist'] = $oGC->GCDistance($geo[0]['lat'],$geo[0]['lon'],$geo[$i]['lat'],$geo[$i]['lon']);
}
/* order by distance. */
$geo = orderBy($geo, 'dist');
/* connect to closeest server */
$conn_id = ftp_connect($geo[1]['ip']) or die("Couldn't connect to $ftp_server");
ftp_login($conn_id, $geo[1]['user'], $geo[1]['pass']);
$contents = ftp_nlist($conn_id, "tts/incoming");
ftp_close($conn_id);
$localfile = '/user/dac15/tts/incoming/';
$remotefile = '/user/'.$geo[1]['user'].'/tts/incoming/';
/* if file is on the closest server. */
if (find($_GET['name'],$contents)) {
// http://www.awesomephp.com/?Tutorials*16/Download-file-with-resume,-stream-and-speed-options.html
downloadFile($remotefile,$_GET['name'],900,false);
}else{
/* push file to the closest server requesting server.*/
$ch = curl_init();
$fp = fopen($localfile.$_GET['name'], 'r');
curl_setopt($ch, CURLOPT_URL, 'ftp://'.$geo[1]['user'].':'.$geo[1]['pass'].'@'.$geo[1]['ip'].'/'.$remotefile.$_GET['name']);
curl_setopt($ch, CURLOPT_UPLOAD, 1);
curl_setopt($ch, CURLOPT_INFILE, $fp);
curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile.$_GET['name']));
curl_exec($ch);
$error_no = curl_errno($ch);
curl_close($ch);
if ($error_no == 0) {
$error = 'File uploaded succesfully.';
$mailto[0][0] = $_SERVER['HTTP_EMAIL'];
$mailto[0][1] = $_SERVER['HTTP_FNAME'];
$mailto[0][2] = $_SERVER['HTTP_LNAME'];
$content="
Dear ".$mailto[0][1]."
<br>
Your file is ready for download. Go to this link in your preferred web browser.
";
// http://www.xpertmailer.com/
if (smtp_mail($mailto,"DoD File Download",$content)) { Return 1; }
}else{
$error = 'File upload error.';
echo $error;
}
}
die;
}
function orderBy($data, $field, $reverse_sort='') {
$code = "return strnatcmp(\$a['$field'], \$b['$field']);";
usort($data, create_function('$a,$b', $code));
if ($reverse_sort==1) { krsort($data); }
return $data;
}
Good Job! I'll have to give you a "Thank You!" :-)
[QUOTE=cesarcesar;170181]SOLVED -
Thanks for everyones help. I am posting my full function for those who may find it useful. To recap, this function "try's" to simulate Akamai's service.
1. GeoLoc user.
2. GeoLoc all servers
3. Calculate distance between user and each server.
4. Sort by distance.
5. Check to see if file is at closest server. If so download.
6. If file not at closest server, copy file from mother server to server missing file. Then
I have another question to go along with this post. Once the cURL transfer is started on my 1GB file the screen just sits white while doing its thing (page load bar crawls). Is there a way to finish loading the page while the transfer is in process. This way the user doesn't have to wait for the transfer to finish, they are just there to make it happen. (they are not downloading at the time so it would be okay for them to continue). Thanks.
/**
* Takes a needle and haystack (just like in_array()) and does a wildcard search on it's values.
*
* @param string $string Needle to find
* @param array $array Haystack to look through
* @result array Returns the elements that the $string was found in
*/
function find ($string, $array = array ())
{
foreach ($array as $key => $value) {
unset ($array[$key]);
if (strpos($value, $string) !== false) {
$array[$key] = $value;
}
}
return $array;
}
Linear Mode
