Archive for the 'knowledge base' Category

How to use Wget

Published
by
dataninja
on January 5, 2009
in how to, knowledge base, open source, software and tools
. 0 Comments

Consider this to be a crash course on How To use Wget. Wget is by far the best (intelligent) download manager ever created for the common client server protocols on the web (HTTP, HTTPS, FTP). The amount of configuration options offered by Wget is so immense that the use of the command line may be necessary in some cases. Here is a list of useful Wget commands. Ports of Wget for other platforms exist, if you want to see more configuration option open the terminal and use.

wget --help

Basic use

The simplest command Wget can offer. This will download the the file movie.mp4 from domain.com, notice how the the protocol HTTP had to specified can be changed for FTP.

wget http://domain.com/movie.mp4

Resume interrupted download

Let’s say you are downloading a 2 GB file then Internet connection dies for some reason Wget only managed to download 1.5 GB out of the 2 GB. The -c option can be used to tell Wget to resume the partially downloaded file, no need to download again.

wget -c http://domain.com/bigfile.iso

Mirror a site

Mirror or make a copy of a web site with the -m option. Self explanatory.

wget -m http://domain.com

Bandwidth throttling

Not everybody has FIOS (not me) for those who for example share a DSL or ISDN line are some what limited in the amount of bandwidth they can spare. Wget provides a way to limit the amount of bandwidth used, basically you can limit how much bandwidth it can use.

wget --limit-rate=20k http://domain.com/movie.mp4

Increase re-tries

When dealing with an Internet connection that suffers from high latency resulting in long download times the downloads will likely fail to complete. Instead of letting the unknown happen raise the default amount of retries (20) to say 30. Hopefully this will give you a higher margin to complete the download.

wget --tries=45 http://domain.com/movie.mp4

Disguised as a web browser

Some sites will not allow download managers alone to retrieve the files stored on their server, instead they expect to see a web browser like Mozilla or IE. Well this minor obstacle can be overcome by telling Wget to disguised it self as Mozilla to retrieve the file.

wget -c -U Mozilla http://domain.com/movie.mp4

Download files from FTP server

To download the same file from a FTP server just change the protocol when issuing the Wget command.

wget ftp://domain.com/movie.mp4

Let’s say the site happens to be password protected, Wget allows the user to login with credentials.

wget ftp://username:password@domain.com/path/to/file

Feel free to mix and match the options above nobody got any where by refusing to experiment.

Interesting Links
http://en.wikipedia.org/wiki/Wget

If you find any of the material on this site useful please take a moment to make a small donation.

Recover deleted or formated data with PhotoRec

Published
by
dataninja
on January 3, 2009
in how to, knowledge base, open source, software and tools
. 0 Comments

Last updated : 01/04/09

Accidents happen for example a Flash drive that might have been accidentally formatted or certain files erased from a hard drive. The good news is that as long as the data is not overwritten there is a high margin for a successful recovery. After all the recycle bin is not a secure way of destroying data by any standards.

For this article I will be using PhotoRec by CG Security, Photorec has many advantages for example it’s open source, cross platform, configurable and is relatively light while operating. The entire operation will take some time depending in size of the drive to be inspected. Do not expect the files to recovered with the original file name, instead Photorec will assign them a number. The output data should be stored on a second drive, don’t even think about about using the same drive.

The recovery technique I use is called Data Carving, I’ll let the collective knowledge explain what it means.

Data Carving is a data recovery technique that allows for data with no file system allocation information to be extracted by identifying sectors and clusters belonging to the file. Data Carving usually searches through raw sectors looking for specific desired file signatures. The fact that there is no allocation information means that the investigator must specify a block size of data to carve out upon finding a matching file signature. This presents the challenge that the beginning of the file is still present and that there is (depending on how common the file signature is) a risk of many false hits. Also, data carving requires that the files recovered be located in sequential sectors (rather than fragmented) as there is no allocation information to point to fragmented file portions. This method can be time and resource intensive.

Excerpt from Wikipedia

Download PhotoRec from CGSecurtity

PhotoRec is part of the TestDisk Suite.
http://www.cgsecurity.org/wiki/PhotoRec

Step 1

After PhotoRec starts you will be presented with all the drives PhotoRec was able to detect. For demonstration purposes I will be using a 2 GB flash drive.

Disk /dev/sdb - 2097 MB / 1999 MiB (RO) - OCZ ET1208AD

And press enter to proceed.

Step 2

Select the partition table in my case it’s.

[ Intel ] intel/PC partition

Step 3

In this case I want to inspect the whole disk for data.

D No partition 0 0 1 254 245 54 4095999 [Whole disk]

Step 4

If you noticed in the bottom of the terminal you are also presented with the following options.

[Options] [File Opt]

You can select [Options] or [File Opt] to see the available file extensions that PhotoRec can recover, you can also unchecked the file extensions you don’t want to recover. Or configure how persistent PhotoRec should be.

Step 5

Now specify the file system on which PhotoRec will attempt to recover the data from. My USB drive was formatted to FAT32.

[ Other ] FAT/NTFS/HFS+/ReiserFS/…

Step 6

PhotoRec is now presenting you with the option of choosing the default output directory /home/user or your own directory. You might want to create a directory just for the recovery which is usually spread across several directories.

Step 7

After selecting the output directory the recovery process will start, PhotoRec will let you know of the remaining time and number of files found.

The recovery process is done. All recovered files are enumerated.

Output

Example of the recovery, all data is spread across multiple directories. The recovered files are renamed.

Not all is lost sometimes most of the data can be recovered. Keep in mind some media files may be beyond recovery. Always back up your data and there will be no need for a recovery, otherwise I hope this article helps.

http://en.wikipedia.org/wiki/Data_recovery

If you find any of the material on this site useful please take a moment to make a small donation.

Open source tools I can’t work without

Published
by
dataninja
on January 3, 2009
in knowledge base, open source, software and tools
. 0 Comments

Whether it’s for recovery or day to day activities open source has a lot to offer. From simple entertainment software to data recovery there are a lot of options available in the Internet. Here is a list of what I commonly use to get the job done, if you have any recommendation feel free to comment.


Testdisk

Sometimes accidents happen and partitions are deleted or the system may become un-bootable for some reason. Useful for those nasty Windows viruses which may damage the partition table.


PhotoRec

PhotoRec was design to recover data from storage media like hard disk and Flash drives alike that may have been accidentally formated or deleted. PhotoRec aims to recover common formats like PDF, HTML MP3 to name a few. When doing a recovery you might only want to recover the actual files that contain the data not the software it used to run on.


DBAN

By far the best tool for those who want to re-use hard drives which may or may not contain sensitive data. Instead of just erasing data DBAN rewrites random data generated by Mersenne twister or ISAAC on the hard drive multiple times. How secure it is?, well it holds several official certifications from government agencies.


Ophcrack

Useful for when the user forgets his or her password and has no other way of obtaining the password to the system. It beats other commercial tools I have used previously.


GParted

For those who boot into multiple operating systems and need to re-size the primary partition or have to re-size the partition where Windows resides. Some PC manufacturers do not provide an actual CD/DVD instead the OS is install in to a partition.


UNetbootin

Some tools are offered in the form of an ISO for a reason meaning they have to burned in to a CD/DVD. UNetbootin allows a USB drive to become bootable reducing the amount of CD/DVD’s I have to burn (waste). UNetbootin removes the hassle of having to make the ISO bootable, just point to the ISO and UNetbootin does the rest.

If you find any of the material on this site useful please take a moment to make a small donation.