Ever since I joined Hostgator.com, I’ve been learning a lot of Linux in the hopes that I switch my career into Linux. Hopefully Forensics related.
So this new dilemma I had was to download a website for offline browsing. I went on the hunt for an offline file browser for Linux…. I found that I could use wget to mirror a whole website.
For example, I want to make a copy of blackberrysimunlockcode.com, Here’s how:
wget -m http://blackberrysimunlockcode.com
Here the -m option is telling wget to mirror the website. This is the basic command. But say I need some advanced options. What do I do?
I was trying to get all the script files off of a website to save for later learning and all it was downloading was the index.html and robots.txt
The robots.txt file was blocking user agent wget. To confirm this I used the debug option in wget:
wget -m -d http://blackberrysimunlockcode.com
You’ll get something like:
Not following http://blackberrysimunlockcode.com/privacy.shtml because robots.txt forbids it.
Rejecting path sh/eg/talk.sh.txt because of rule `sh’
no-follow in index.html
I tried using the option –user-agent “Mozilla” ….. no luck
I tried adding the following in .wgetrc :
## Local settings (for a user to set in his $HOME/.wgetrc). It is
## *highly* undesirable to put these settings in the global file, since
## they are potentially dangerous to “normal” users.
## Even when setting up your own ~/.wgetrc, you should know what you
## are doing before doing so.
header = Accept-Language: en-us,en;q=0.5
header = Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
header = Accept-Encoding: gzip,deflate
header = Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
header = Keep-Alive: 300
user_agent = Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:220.127.116.11) Gecko/20070725 Firefox/18.104.22.168
referer = http://www.google.com
…Still no luck.
The trick is to use option -e robots=off
So my new command became:
wget -m -k -e robots=off -w 2 --random-wait -U "Mozilla" -np http://blackberrysimunlockcode.com
Heres what the options do:
-m mirrors website
-k fix links so you don’t get directed to www.blackberrysimunlockcode.com/sh/eg instead of /sh/eg (relative vs absolute)
-e executes command robots=off
-w 2 sets wait time as 2 seconds so you don’t overload server and get ip blocked
–random-wait can be random in 2 secon increments
-U sets user agent
-np no parent, so if the current subdirectory/page links to parent pages, it might crawl whole website
The Nexus one was rooted before it even got released. Paul at modaco forums showed us how to perform this hack. I’ll try to simplify the method here.
1. First unlock your bootloader if you have a T-mobile Nexus One (this voids the warranty) or get an unlocked version from Google for like $550:
a. download fastboot
b. open terminal or command line and type
fastboot-windows oem unlock
./fastboot-mac oem unlock
./fastboot-linux oem unlock
2. Rewrite the Bootloader to obtain root privileges:
a. Download either SuperBoot or InsecureBoot and extract. (The difference is that with the insecure boot you won’t make any changes to the system and you would have to launch su manually)
b. Put the device in bootloader mode:
– Turn off the phone and restart holding the trackball.
c. Flash the bootloader:
– Windows = double click the windows bat file
– Mac = open terminal and cd to the directory containing the file. type
chmod +x install-superboot-mac.sh
– Linux = open terminal and cd to the directory containing the file. type
chmod +x install-superboot-linux.sh
Now you should have root!
Pic courtesy of Stericson.
Last time I backed up my website with rsync and ssh, but on my new host they disabled SSH. Rsync does not work over ftp. I do not want to do incremental backups with delta files like is done with rdiff-backup or duplicity. I need to have an exact mirror of my site. But remember that your sql databases won’t be backed up.
Curlftpfs is the key! In this tutorial I will show you how to backup from one server to a backup location which can be your hard drive, another web host, dropbox folder, Box.net via webdav, etc.
Download curlftpfs, rsync and ncftp:
sudo apt-get install curlftpfs ncftp rsync
make directories to mount your ftp server:
sudo mkdir /media/hydtechblog
sudo mkdir /media/hydtechbackupserver
edit fstab to mount the ftp servers using curlftpfs:
sudo gedit /etc/fstab
add the lines and modify them according to your server:
curlftpfs#username:email@example.com /media/hydtechblog fuse rw,allow_other,uid=root 0 0
curlftpfs#username:password@hydtechbackupserver /media/hydtechbackupserver fuse rw,allow_other,uid=root 0 0
One thing to remember is that these two will not mount automatically because when the computer restarts, the fstab is done while you are not connected to the network. To fix this we can just add the mount commands in our crontab.
sudo crontab -e
enter the following lines and modify accordingly:
00 09 * * * mount /media/hydtechblog
00 09 * * * mount /media/hydtechbackup
01 09 * * * rsync -avz –rsync-path=/usr/bin/rsync /media/hydtechblog/public_html /media/hydtechbackup/public_html
ctrl + o to write and ctrl + x to save
This will tell cron to mount the folders at 9:00 am and start rsync at 9:01 am. You can replace the backup location to another folder on your hard drive or your dropbox or ubuntu one folder. You can also mount box.net with webdav and use this method.
For encrypted incremental backups checkout duplicity, it also works with webdav and ftp.
In this tutorial I will be showing you how to Quad boot your macbook with OS X, Windows Vista, Windows XP, and Ubuntu Linux. You can leave out the OSes you don’t want or replace them with something else. This is just a general but detailed guide for noobz!
We will be installing OS X from scratch so all your setting will be deleted. You might want to back up the data with Time machine or another backup software. Insert the OS X DVD and reboot the macbook holding down the C key.
Select your Language.
Before we install the OSes, we will partition the hard drive into 5 partitions. Go to Utilities and click on Disk Utility.
Select the Hard drive on the left. Mine has a 120GB drive.
Click on the partition tab and select 5 Partitions from the Volume Scheme. Label them as follows:
1. Vista (MS-DOS FAT and atleast 10GB)
2. Storage (MS-DOS FAT and however much space you want as a storage location)
3. XP (MS-DOS FAT and atleast 7GB)
4. OSX (Extended Journaled and atleast 20GB)
5. Free space (Chose from the drop down. will be used for our Ubuntu Linux)
Click on Options and make sure GUID Partition Table is selected. Hit Apply
Now we are going to Install our first OS. Select the OSX partition and continue.
You can skip the check if you know the DVD is in good condition and without scratches.
Your Installing your first OS. excited?
Be patient. This will take a while.
Restart your computer.
It will ask you for the second Disc. Insert the second disc.
Installation Complete!! now pat yourself on the back.
Once you setup your OSX with your settings, open up the safari web browser and goto http://refit.sourceforge.net/ and download the Mac disk image
once the download finishes, double click the image and then on the installer to install it.
After rEFIt is installed, insert the VISTA disc and reboot holding the C key.
Accept the terms.
Select the Vista partition and format with NTFS before continuing with Install.
After restarting, you will see the rEFIt GUI. select the second partition to boot into VISTA.
Goto My computer.
select the CD/DVD drive.
Eject the CD and insert the OSX disc.
Select Open folder.
Go into Boot Camp folder.
Continue to Install the BootCamp drivers.
Select No to restart. We need to mark our XP partition as active first.
Goto Control Panel.
System and maintenance
Click ‘Create and format hard disk partitions’ under Administrative tools.
right click the XP partition and formate with NTFS.
after formatting is complete, right click the XP partition and mark as active.
Hell yes. Now eject the OSX disc and insert the XP disc.
Upon entering rEFIt, select boot from CD.
Select partition 4
Let it do its thang.
upon restart, Boot from partition 4.
Awesome!! Now you have 3 OSes. Take a break.
In XP, goto my computer and eject the CD. Insert the OSX disc.
Install Boot Camp drivers.
Select no to the restart. Eject the OSX disc and insert the Ubuntu CD. Now restart
Boot from CD.
Pick your timezone.
Select USA – Macintosh for the keyboard.
Select the free space and click on new partition.
Now we will make a swap space for Linux.
Size: more than your RAM to be able to hibernate. My RAM is 2GB so I choose 2001MB.
Use as: Swap area
Create another partition.
Size: Use the remaining disk space.
Use as: Ext3
Mount point: /
type in your username and password. Don’t forget your password.
nothing to migrate. so leave this blank.
Click on advanced and in the drop down box type in (hd0,2)
Ubuntu will ask you to remove the disc and press ENTER.
You will come back to rEFIt
Start the partitioning tool.
Type Y to update the MBR.
Boot from the Linux partition
Boot into Vista.
You should get an error. If you get this error, Follow the directions and enter the Vista CD. restart the computer and select Boot from CD from the rEFIt menu.
repair and restart.
now we want our Ubuntu to start automatically without waiting for a timeout when we select the Ubuntu partition, instead of showing us the Grub menu. To do this, boot into ubuntu and open a terminal shell.
type sudo gedit /boot/grub/menu.lst
enter your password.
you will be presented with the grub configuration file in a text editor. change the timeout to 0.
Enjoy your QUAD BOOT MACBOOK !!!!