1. Welcome to PHCorner Forums. Take a moment to Sign up and gain unlimited access and extra privileges that guests are not entitled to, such as:

    All that and more! Registration is quick, simple and absolutely free. Join our community today!

Tutorial How do I mirror another website

Discussion in 'Web & Internet' started by DCCS, Mar 6, 2015.

  1. DCCS

    DCCS Forum Veteran Established

    Downloading a copy of an entire website ("mirroring") is a tricky business, because modern websites are so often built with PHP, ASP, CGI and other dynamic technologies that constantly update every page and often produce Please or Register to view links that are only used once. And that can make the site appear infinitely large to website mirroring software.

    Also, mirroring the web pages your browser sees on a site doesn't mean you'll get the "dynamic" behavior. if a website displays the current weather in Cleveland, mirroring the current pages will only get you a frozen "snapshot" of the weather on that particular day.

    That said, though, there are tools available that will help you mirror basic websites that don't have these problems. The best-known of these is GNU wget, a free, open-source tool that can easily fetch an entire website with a single command. wget is not the friendliest tool in the world, but boy does it work!

    Mirroring a website on Windows
    If you are running Windows, I recommend Tech Knight's Please or Register to view links. Tech Knight offers step-by-step instructions to download and use the wgetsoftware on Windows. Please or Register to view links and compile your own copy from source.
    Once you have wget installed correctly, the command line to mirror a website is:

    wget -m -k -K -E Please or Register to view links

    See man wget or wget --help | more for a detailed explanation of each option.

    If this command seems to run forever, there may be parts of the site that generate an infinite series of different URLs. You can combat this in many ways, the simplest being to use the -l option to specify how many links "away" from the home page wget should travel. For instance, -l 3 will refuse to download pages more than three clicks away from the home page. You'll have to experiment with different values for -l. Consult man wget for additional workarounds.

    Note: some web servers may be set up to "punish" users who download too much, too fast. If you're not careful, using tools like wget could get your IP address banned from the site. You can avoid this problem by using the -w option to specify a delay, in seconds, between page downloads. Usually, this will prevent the web server from viewing your behavior as unacceptable. But your mileage may vary!
    Mirroring a website on MacOS X
    Like Linux, MacOS X is a version of Unix. However, wget isn't standard equipment in all versions of MacOS X. If you receive an error message when you try the wget --help command at the MacOS X "Terminal" prompt, you can fetchwget from the Please or Register to view links, which also offers "Simple wget," a user-friendly front end to wget. Most of the site is in Japanese, so some patience is necessary in picking your way through!
    Of course, you can also install the developer tools from your MacOS X system CD (if you have not already done so) and then visit the Please or Register to view links to build and install wget from source code.

    Once you have the command line version of wget for MacOS X installed, just follow my Please or Register to view links at the MacOS X Terminal prompt.

    Offering Your Mirror To The World
    Publicly mirroring someone else's website without their permission is a violation of copyright law. Don't do that.
    If you have received their permission, it's easy to offer your mirror to the world. Just use the wget command to download it to a directory inside your own website's space. This is much easier if you have command line access to your own web server so that you can run wget there directly. But you can also upload the mirrored site to your server by dragging and dropping it to your usual file transfer program after wget is finished.

    If you do offer a mirror of another site, make sure you link to the original and explain to users that this is a mirror and not the original. Also be sure to keep your mirror up to date. And once again, get the original site's permission first!

    Example of website that i already mirrored.
    Please or Register to view links

    12345mahalkita likes this.
  2. derx

    derx El Vampiro Staff Member Moderator

    Galing galing!!! Thanks boss! :geek: (y) :cat:
  3. DCCS

    DCCS Forum Veteran Established

    Salamat bro :)
  4. is it okay if share some of the good stuff in here to me website :)
  5. It seems cool! pero bawal kase yan eh katakot
  6. DCCS

    DCCS Forum Veteran Established

    Masarap ang bawal. :hilarious:
  7. theme lang ang ma copy ? o pati yung post ? :)
  8. DCCS

    DCCS Forum Veteran Established

    Pati post.
  9. oy ayos to ah :) ma try nga :)
  10. Teka pano ba gamitin yong "wget -m -k -K -E Please or Register to view links"