Retrieving external dynamic pages with variables through PHP - GoFuckYourself.com

Odin88 · 08-20-2003, 11:44 PM

I have encountered a bump on a project I am currently working on. The project requires a script to retrieve the content of an external page with variables for further processing, and I can't seem to get it to work even when using FSockOpen. This may, or should, be a failure on my part, but I looked the issue up in Google and found that others have had the same problem. Among other things, I found some interesting discussion forum threads, but the answers (if you can call it that) was rather vague and speculative and lead me no closer to a solution.

I know for a fact that what I am seeking to do is possible, some way or the other, but this is the first time I have had to retrieve content from external pages with variables. Any input would be highly appreciated.

Danke schön.

igneus · 08-21-2003, 12:02 AM

Not sure what you mean by "with variables" .. but for just saving the contents of a webpage to a variable, have you tried "file_get_contents"? Probably the easiest solution.

Could also use fopen and all that crap too

Nathan · 08-21-2003, 12:08 AM

odin:

simple, do the following:

the external page should look something like this:

PHP Code:


			
&lt;? echo "<?"; ?&gt;

$var = "eek";

$foo = "bar";

$somemore = 1;

&lt;? echo "?>"; ?&gt;

Lets pretend that page would be http://www.fuckyoutoo.com/some.php

you would simply do:
include("http://www.fuckyoutoo.com/some.php"); in your main file that needs the vars.

You can of course also not use those strange echos and print the <? ?> directly in the file, but you have to make sure that the server that file is on will not parse it for php since it'll of course be empty for the include() then.

IMP^or^SNiTL.e · 08-21-2003, 12:11 AM

Odin88 · 08-21-2003, 12:36 AM

Quote:

Originally posted by Nathan
odin:

simple, do the following:

the external page should look something like this:

PHP Code:

<? echo "<?"; ?> $var = "eek"; $foo = "bar"; $somemore = 1; <? echo "?>"; ?>

Lets pretend that page would be http://www.fuckyoutoo.com/some.php

you would simply do:
include("http://www.fuckyoutoo.com/some.php"); in your main file that needs the vars.

You can of course also not use those strange echos and print the <? ?> directly in the file, but you have to make sure that the server that file is on will not parse it for php since it'll of course be empty for the include() then.

You do not quite understand what I am trying to do, but that is my fault I suppose. "Retrieving dynamic pages with variables" can be interpreted many ways. What I am trying to do is actually to fetch the output of dynamic pages such as http://www.google.com/search?hl=en&i...=Google+Search

I appreciate your time and effort though.

Nathan · 08-21-2003, 12:59 AM

Hey.. emm.

whats the problem about doing that? <G>

$lines = file("http://www.somepage.com?with=vars&foo=bar");

that doesn't work for you?

Odin88 · 08-21-2003, 01:20 AM

Quote:

Originally posted by Nathan
Hey.. emm.

whats the problem about doing that? <G>

$lines = file("http://www.somepage.com?with=vars&foo=bar");

that doesn't work for you?

There's no need to be condescending. That doesn't work either.

artware · 08-21-2003, 01:41 AM

Is PHP set up correctly to allow the fopen wrappers? That could be the reason why it doesn't work..?

Read about it here.

rudeboi · 08-21-2003, 03:14 AM

Quote:

Originally posted by Odin88

You do not quite understand what I am trying to do, but that is my fault I suppose. "Retrieving dynamic pages with variables" can be interpreted many ways. What I am trying to do is actually to fetch the output of dynamic pages such as http://www.google.com/search?hl=en&i...=Google+Search

I appreciate your time and effort though.

Read up on the google api, and rss.. there are php scripts out
there to let you use the google api as rss/rdf feeds.

I've got one that workds with news.google.com if you need one as an example.

Brian911 · 08-21-2003, 04:17 AM

just use wget

fiveyes · 08-21-2003, 04:34 AM

Get a Google Api account and then check out http://www.jappc.com/google

KC · 08-21-2003, 04:41 AM

Yep, Google API is the way to do it..

http://www.google.com/apis/api_faq.html

KC · 08-21-2003, 04:47 AM

If Google isn't the "dynamic pages" you want to pull variables from, then you'll probably need to do some sort of REG/EX to pull the correct values.

IMP^or^SNiTL.e · 08-21-2003, 04:52 AM

KC · 08-21-2003, 04:57 AM

Quote:

Originally posted by IMP^or^SNiTL.e
sexy

How much swoit are you raking in a day with that thing?

venturi · 08-21-2003, 05:47 AM

The only way you can "trap" these variables is if you parse the URI in your php script. If they are hidden or session variables you are basically screwed unless you already know the variable names associated with the page you are attempting to 'scan'.

People that have accomplished what I think you are trying to do have been given explicit permission from the sites they are parsing and have been granted access to some sort of an API that said site(s) have available.

I may not be Completely accurate here but I think that using any kind of TCP socket connection to said web page will not garner you the info you wish to gather.

rudeboi · 08-21-2003, 05:35 PM

Also, look into CURL .. you can do a lot of
page scraping type work with CURL .. or just go to CPAN.org and search for page scrape and do your page scraping in perl

08-20-2003, 11:44 PM	#1
Odin88 Confirmed User Join Date: Jul 2003 Location: au Posts: 3,267	Retrieving external dynamic pages with variables through PHP I have encountered a bump on a project I am currently working on. The project requires a script to retrieve the content of an external page with variables for further processing, and I can't seem to get it to work even when using FSockOpen. This may, or should, be a failure on my part, but I looked the issue up in Google and found that others have had the same problem. Among other things, I found some interesting discussion forum threads, but the answers (if you can call it that) was rather vague and speculative and lead me no closer to a solution. I know for a fact that what I am seeking to do is possible, some way or the other, but this is the first time I have had to retrieve content from external pages with variables. Any input would be highly appreciated. Danke schön.

08-21-2003, 12:02 AM	#2
igneus Registered User Join Date: Jul 2003 Posts: 77	Not sure what you mean by "with variables" .. but for just saving the contents of a webpage to a variable, have you tried "file_get_contents"? Probably the easiest solution. Could also use fopen and all that crap too __________________ icq: 201430673

08-21-2003, 12:08 AM	#3
Nathan Confirmed User Industry Role: Join Date: Jul 2003 Posts: 3,108	odin: simple, do the following: the external page should look something like this: PHP Code: `<? echo "<?"; ?> $var = "eek"; $foo = "bar"; $somemore = 1; <? echo "?>"; ?>` Lets pretend that page would be http://www.fuckyoutoo.com/some.php you would simply do: include("http://www.fuckyoutoo.com/some.php"); in your main file that needs the vars. You can of course also not use those strange echos and print the <? ?> directly in the file, but you have to make sure that the server that file is on will not parse it for php since it'll of course be empty for the include() then. __________________ "Think about it a little more and you'll agree with me, because you're smart and I'm right." - Charlie Munger Last edited by Nathan; 08-21-2003 at 12:12 AM..

08-21-2003, 12:11 AM	#4
IMP^or^SNiTL.e A/S/L .. I don't names. Industry Role: Join Date: Aug 2003 Posts: 1,177	bah

08-21-2003, 12:59 AM	#6
Nathan Confirmed User Industry Role: Join Date: Jul 2003 Posts: 3,108	Hey.. emm. whats the problem about doing that? <G> $lines = file("http://www.somepage.com?with=vars&foo=bar"); that doesn't work for you? __________________ "Think about it a little more and you'll agree with me, because you're smart and I'm right." - Charlie Munger

08-21-2003, 01:41 AM	#8
artware Confirmed User Join Date: Oct 2002 Location: the Netherlands Posts: 268	Is PHP set up correctly to allow the fopen wrappers? That could be the reason why it doesn't work..? Read about it here.

08-21-2003, 04:17 AM	#10
Brian911 Confirmed User Join Date: Dec 2001 Posts: 654	just use wget __________________ - Brian [ a directive occured while processing this error ]

08-21-2003, 04:34 AM	#11
fiveyes Confirmed User Join Date: Aug 2001 Location: New Orleans Posts: 1,680	Get a Google Api account and then check out http://www.jappc.com/google __________________ <CENTER><A HREF="http://www.hot-off-bourbon.com/" target="_blank"><IMG SRC="http://www.hot-off-bourbon.com/images/hob-logosmall.jpg" border="0"></A> <FONT face="Comic Sans MS" SIZE="-1"><I>Mardi Gras, Spring Break, Wet-T, Night Club Action, UpSkirt, Oil Wrestling, Voyeur</I></FONT></CENTER>

08-21-2003, 04:41 AM	#12
KC Confirmed User Industry Role: Join Date: Jan 1995 Posts: 2,417	Yep, Google API is the way to do it.. http://www.google.com/apis/api_faq.html __________________ Jupiter Hosting, Inc. Vice President, Business Development kc (AT) jupiterhosting.com

08-21-2003, 04:47 AM	#13
KC Confirmed User Industry Role: Join Date: Jan 1995 Posts: 2,417	If Google isn't the "dynamic pages" you want to pull variables from, then you'll probably need to do some sort of REG/EX to pull the correct values. __________________ Jupiter Hosting, Inc. Vice President, Business Development kc (AT) jupiterhosting.com

08-21-2003, 04:52 AM	#14
IMP^or^SNiTL.e A/S/L .. I don't names. Industry Role: Join Date: Aug 2003 Posts: 1,177	sexy

08-21-2003, 05:47 AM	#16
venturi Confirmed User Join Date: Aug 2001 Location: An Oasis atop a High Desert Mesa Posts: 386	The only way you can "trap" these variables is if you parse the URI in your php script. If they are hidden or session variables you are basically screwed unless you already know the variable names associated with the page you are attempting to 'scan'. People that have accomplished what I think you are trying to do have been given explicit permission from the sites they are parsing and have been granted access to some sort of an API that said site(s) have available. I may not be Completely accurate here but I think that using any kind of TCP socket connection to said web page will not garner you the info you wish to gather. __________________ I'm not a porn monger. I'm a porn ... STYLIST! Buildit Cheap, build it Fast, Build it Right. - Pick Two.

08-21-2003, 05:35 PM	#17
rudeboi Confirmed User Join Date: Jun 2003 Location: "Sunny" San Francisco Posts: 262	Also, look into CURL .. you can do a lot of page scraping type work with CURL .. or just go to CPAN.org and search for page scrape and do your page scraping in perl __________________ ---------------------------------------------- Systems Architect http://www.bitpusher.com/