Save a complete webpage *and the pages it links to*

[djr33]

I'm a member of another forum, and my PM inbox is nearly full. It uses outdated phpBB software with no "export" option, and saving these messages would require manually saving every single one, and the result would be an unorganized mess.

I've been wondering if there's a way to save a webpage *and the pages it links to*, so that when I click "file>save" it actually saves all of those links. Of course technically it probably links to all of the pages on the internet, if we follow each link, then each link at that link, and so forth. But I could either set it to just one-level, or to a limited directory, or whatever.

Another way of phrasing this is whether it's possible to save all files in a certain directory via HTTP rather than FTP, assuming all of the links are known/knowable. (I'm not trying to access hidden files.) Technically, this would include PHP-generated dynamic pages, such as those at a forum.

Of course this could potentially be used for bad purposes (stealing an entire forum), but let's focus on the good uses.

I know I could write this myself in PHP, but it would be a pain, and I'd probably be better off just saving all of the messages manually.

Any ideas?

[jscheuer1]

I think you can set something like that up on a server using PHP and an IE browser, but it's complicated.

What is done, if your server is Windows based, it should have IE installed on it or you can install it if it's not, you can then use PHP to drop to the OS and run the browser, navigate to the links one by one by 'reading' them off of the page, capturing the each page to a file as you go. You could even concat to one file.

But it just occurs to me typing this, that it might be easier to get the links using a file_get_contents on the main index page for your mailbox and parsing that to get the links you're interested in, then do a loop on those to file_get_contents on each that writes them to a single file, something like (for inside the loop):

file_put_contents('mymail.txt', file_get_contents('protocol://domain.com/path/thispage.ext'), FILE_APPEND); 

You could replace file_get_contents('protocol://domain.com/path/thispage.ext') in the above with a subroutine to get just the text part of the message if you like.

Just a general idea.

[djr33]

I could write it myself (using file_get_contents(), or something along those lines), but this would be somewhat complicated because:
1. I'd rather just have a program that does it for me (if such a program exists-- I've found a couple that claim to do things like that on Google, although I'm not sure how well they work. I'm still looking around).
2. This particular case involves cookies (I need to be logged in), so that makes using PHP a little trickier. Not impossible, just trickier.
3. I'd like to have the links automatically set. It wouldn't be too hard to just loop through and save everything, but I'd like to have, for example, the timestamps from the private messages still useful for organizing them.

[ajfmrf]

The firefox add on "down them all" could be something like what you want.It will down load links on a webpage and images...

http://www.downthemall.net/

I wonder if it could help you somehow Daniel?