Interesting and useful shell commands

                                    pdf_icon.png Download this article as a single PDF document 

Introduction

Here's a list of some cool *nix commands and built in shell functions.
If you cannot find a particular command on your system, you may probably install it yourself.


The magic

# date --set="$(ssh user@server date)"


# ssh root@remote.host "rpm -qa" | xargs yum -y install


if [ -z "$@" ]; then
  if [ "$(pgrep -fx "/bin/bash $0")" != "$$" ]; then
    echo "ERROR: another instance of '$(basename $0)' is already running"
    exit 1
  fi
else
  if [ "$(pgrep -fx "/bin/bash $0 $@")" != "$$" ]; then
    echo "ERROR: another instance of '$(basename $0) $@' is already running"
    exit 1
  fi
fi


# export HISTTIMEFORMAT="%F %T "


# kill -9 $$


# tail -f /var/log/messages | ccze -A


# lsof -i tcp:80

Piping the output through ccze will give you an even more pleasant output:

# lsof -i tcp:80|ccze -A


# free && sync && echo 3 > /proc/sys/vm/drop_caches && free


# find . -name "*.[ch]" -exec grep -i -H "search pharse" {} \;


# vim scp://username@host//path/to/somefile


# netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'


 # strings /dev/urandom | grep -o '[[:alnum:]]' | head -n 30 | tr -d '\n'; echo


# ss -p
# ss -p | grep STA  <- for established sockets only
# ss -p | cut -f2 -sd\" <- just process names


# find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate


# rename 'y/ /_/' *


# ssh-keygen -R <the_offending_host>


# bind -P


# sudo touch /forcefsck


# for code in {0..255}; do echo -e "\e[38;05;${code}m $code: Test"; done


Run 'nc yourip 5000', 'nc yourip 5001' or 'nc yourip 5002' elsewhere will produce an exact same mirror of your shell.
This is handy when you want to show someone else some amazing stuff in your shell without giving them control over it.

# script -qf | tee >(nc -kl 5000) >(nc -kl 5001) >(nc -kl 5002)


# sed -n '10,20p' <filename>


# sed -i 8d ~/.ssh/known_hosts


# mkdir -p a/long/directory/path


# mv filename.{old,new}


# awk '!x[$0]++' <file>


# strace -ff -e trace=write -e write=1,2 -p SOME_PID


# getconf LONG_BIT


# !*
# \foo

e.g. if rm is aliased for 'rm -i', you can escape the alias by prepending a backslash.


Deletes all files in a folder that are NOT *.foo, *.bar or *.baz files.

# rm !(*.foo|*.bar|*.baz)


# history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head


# ps aux | sort -nk +4 | tail


# sudo dd if=/dev/mem | cat | strings


# fuser -k filename


# less +F somelogfile

# nc -v -l 80 < file.ext


# ping -i 60 -a IP_Address


# disown -a && exit
# echo "!!" > foo.sh


# pushd /tmp

Example:

# cd /some/place/in/the/filesystem
# pushd /tmp
# ls -la
# popd  <- returns you back to /some/place/in/the/filesystem


# <ctrl+u> [...] <ctrl+y>

Example:

# sudo vim /root/bin/

... autocomplete doesn't work...

#<ctrl+u> sudo ls /root/bin

Ah! that's the name of the file!

# <ctrl+y> sudo vim /root/bin/    #resume here! 


Use ctrl-d to stop it.

# time read 


# net rpc shutdown -I ipAddressOfWindowsPC -U username%password
# net rpc shutdown -r ipAddressOfWindowsPC -U username%password 
# net rpc abortshutdown ipAddressOfWindowsPC -U username%password


# ctrl-l


# (cd /tmp && ls)


# du -h --max-depth=1


# wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com


# curl ifconfig.me
# curl ifconfig.me/ip -> IP Adress
# curl ifconfig.me/host -> Remote Host
# curl ifconfig.me/ua ->User Agent
# curl ifconfig.me/port -> Port


# sudo cpulimit -p pid -l 50


# grep -rl string_one some_directory/ | xargs sed -i 's|string_one|string_two|g'


# awk '{gsub("foo","bar",$5)}1' file


# tar xfz filename.tar.gz -C PathToDirectory


# timeout 5s COMMAND


# youtube-dl -t --extract-audio --audio-format mp3 YOUTUBE_URL_HERE


Install SSHFS from http://fuse.sourceforge.net/sshfs.html
This will allow you to mount a folder securely over a network.

# sshfs name@server:/path/to/folder /path/to/mount/point


Very useful for rerunning a long command changing some arguments globally.
As opposed to ^foo^bar, which only replaces the first occurrence of foo, this one changes every occurrence.

# !!:gs/foo/bar


Particularly useful if you're mounting different drives, using the following command will allow you to see all the filesystems currently mounted on your computer and their respective specs with the added benefit of nice formatting.

# mount | column -t


Prepending one or more spaces to your command won?t be saved in history.
Useful for secrets such as passwords on the commandline.

# <space>command



Useful for checking if there are differences between local and remote files.

# ssh user@host cat /path/to/remotefile | diff /path/to/localfile -


Makes a partition in ram which is useful if you need a temporary working space as read/write access is fast.
Be aware that anything saved in this partition will be gone after your computer is turned off.

# mount -t tmpfs tmpfs /mnt -o size=1024m


Query Wikipedia by issuing a DNS query for a TXT record.
The TXT record will also include a short URL to the complete corresponding Wikipedia entry.

# dig +short txt <keyword>.wp.dg.cx


The PID will only be printed if you?re holding a root equivalent ID.

# netstat -tlnp


This will output the sound from your microphone port to the ssh target computer?s speaker port.
The sound quality is very bad, so you will hear a lot of hissing.

# dd if=/dev/dsp | ssh -c arcfour -C username@host dd of=/dev/dsp


This is an alternative to cron which allows a one-off task to be scheduled for a certain time.

# echo ?ls -l? | at midnight


# curl -u user:pass -d status=?Tweeting from the shell? http://twitter.com/statuses/update.xml


Now you can acces the website by going to http://localhost:2001/

# ssh -N -L2001:localhost:80 somemachine


If you break your terminal by sending binary data to STDOUT or similar, you can get your terminal back using this command rather than killing and restarting the session.
Note that you often won't be able to see the characters as you type them.

# reset


# ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq /tmp/out.mpg


For when you want to flush all content from a file without removing it.

# > file.txt


To generate the keys use the command ssh-keygen.

# $ssh-copy-id user@host


Next time you are using your shell, try typing ctrl-x e (that is holding control key press x and then e).
The shell will take what you?ve written on the command line thus far and paste it into the editor specified by $EDITOR.

# ctrl-x e


!some_command will search your command history and execute the first command that matches ?some_command?.
If you don?t feel safe doing this put :p on the end to print without executing.
Recommended when running as superuser.

# !some_command:p


The mtr command combines the functionality of the traceroute and ping programs in a single network diagnostic tool.
As mtr starts, it investigates the network connection between the host mtr runs on and HOSTNAME by sending packets with purposly low TTLs.
It continues to send packets with low TTL, noting the response time of the intervening routers.
This allows mtr to print the response percentage and response times of the internet route to HOSTNAME.
A sudden increase in packetloss or response time is often an indication of a bad (or simply over?loaded) link.

# mtr google.com


# cp filename{,.bak}


Really useful for when you have a typo in a previous command. Also, arguments default to empty so if you accidentally run:

# echo ?no typozs?

You can correct it with ^z

# ^foo^bar


# cd -


I often forget to sudo before editing a file I don?t have write permissions on.
When you come to save that file and get the infamous ?E212: Can?t open file for writing?, just issue that vim command in order to save the file without the need to save it to a temp file and then copy it back again.

# :w !sudo tee %


# python -m SimpleHTTPServer


Useful when you forget to use sudo for a command. ?!!? grabs the last run command.

# sudo !!



That's all folks.
Marcin


<comments />
Talk:Interesting and useful shell commands

Useful MySQL commands

                                    pdf_icon.png Download this article as a single PDF document 

Introduction

MySQL is the world?s most popular open source database.
Whether you are a fast growing web property, technology ISV or large enterprise, MySQL can cost-effectively help you deliver high performance, scalable database applications.


The commands

# watch -n 1 mysqladmin --user= --password= processlist


I put this in a cron job to run @ midnight as a lazy back up solution.

# for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done


This will dump a MySQL database over a compressed SSH tunnel and uses it as input to MySQL.
It's a very fast way to migrate a DB to a new server.

# mysqldump --add-drop-table --extended-insert --force --log-error=error.log -uUSER -pPASS OLD_DB_NAME | ssh -C user@newhost "mysql -uUSER -pPASS NEW_DB_NAME"


# mysql --database=dbname -B -N -e "SHOW TABLES"  | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;"}' | mysql --database=dbname &


# ssh user@host 'mysqldump dbname | gzip' > /path/to/backups/db-backup-`date +%Y-%m-%d`.sql.gz


# echo "SELECT * FROM table; " | mysql -u root -p${MYSQLROOTPW} databasename | sed 's/\t/","/g;s/^/"/;s/$/"/;s/\n//g' > outfile.csv


# ssh -CNL 3306:localhost:3306 user@database_server.tld


# echo "SHOW PROCESSLIST\G" | mysql -u root -p | grep "Info:" | awk -F":" '{count[$NF]++}END{for(i in count){printf("%d: %s\n", count[i], i)}}' | sort -n


# mysqldump -u UNAME -p DBNAME TABLENAME> FILENAME


Links

Here is a nice list of MySQL Commands: http://www.pantz.org/software/mysql/mysqlcommands.html



Marcin


<comments />

Malena said ...

<comment date="2013-01-09T02:26:37Z" name="Malena"> perhaps Strato uses different srveers for their different hosting products, the above paths are the standard ones for Linux based srveers I have no idea how the paths would look like on a windows based server though, sorry </comment>

Poor man's VPN tunnel using SSH

                                    pdf_icon.png Download this article as a single PDF document 

"VPN" tunnel using SSH

Introduction

This HowTo explains how to create 2-way SSH tunnel between two locations.
With this you will be able to connect from one location to another as if you had VPN tunnel between them.

Software

All you need is SSH and two hosts running SSH - server and client.


Execution

Create following script:

#!/bin/sh

REMOTE_SERV=11.22.33.44 

while true ; do
        ssh -l your_username -t -g -N -R 22222:127.0.0.1:22 "$REMOTE_SERV" ssh -t -g -N -p 22222 -L 2222:127.0.0.1:22 127.0.0.1
done

And run this script in screen. The while loop should reconnect if the tunnel gets down.
Now you should be able to connect to the computer running the script from the REMOTE_SERV using port 22222.

Cavecats

The tunnel may go down if there is no traffic flow between the nodes.
To solve this issue you can open a screen session, connect to the REMOTE_SERV and run top command.
Or any other process which will periodically send traffic through the tunnel.

That's all folks !


That's all folks. Marcin


<comments />

VPN?? said ...

<comment date="2012-08-16T08:53:44Z" name="VPN??"> ????VPN?????SSH??=?=????????https://www.diyssh.com/?????? http://www.diyssh.com/ </comment>

How to add fonts for X Windows

                                    pdf_icon.png Download this article as a single PDF document 

Introduction.

Hi and welcome to this how-to that explains how to install new fonts for X Windows on a UNIX-like operating system.
It is a simple task to do this through your shell or with the aid of your desktop manager, though i only cover KDE.
Once you've read this paper you'll know how to add fonts for all X-11 applications, like The GIMP and web browsers like Konqueror.
If you don't have any fonts to install yet I'd recommend 1001freefonts.com or try searching google.


Installing fonts manually for all users.

Before you start, you will need root access if you want to add new fonts for all users.
The font's for all users of a system using X Windows are kept in /usr/share/fonts/ , so adding a new font is just a matter of copying your new font to that directory.
You can create a subdirectory in this directory if you like, it helps to keep your fonts more organized, you might create a directory called 'webfonts' or 'gimpfonts', for example.
The command you enter should look something like this :

# mv font.ttf /usr/share/fonts/

And that's all there is to installing a font for all users that are using X Windows, you may need to restart X Windows applications before the fonts can be used but there is no need to restart X Windows itself.


Installing fonts manually for a specific user.

While the font's for all users are kept in /usr/share/fonts the fonts for a specific user are kept in their home directory,
The exact location is ~username/.fonts ( or just '~/.fonts' for your own home directory)
Note: The fonts directory is prefixed with a dot, meaning it is a hidden directory.


Simply copy or move your font file to ~username/.fonts , and your X applications are ready to use your new font. (some applications might require you to restart the application for the new fonts to be visible.)
The command you issue should look something like :

# mv font.ttf ~username/.fonts/


Installing new fonts using KDE.

Maybe you don't like installing new fonts via the console, and would like to do it using a GUI..
KDE and GNOME offer you your alternative, though i only cover KDE.
Enter fonts:/// in your Konqueror address bar and you'll be presented with a "Personal" Directory and a "System" directory.

Fonts for the user you are logged in as are kept in "Personal" and fonts for all users are kept in "System",
It is simply a matter of dragging and dropping your font into either of these directories to install a new font.
Note: You will need to be root to install fonts into the system directory.


Written by Robert Gleeson


<comments />

Useful SVN Commands

                                    pdf_icon.png Download this article as a single PDF document 

Basic SVN commands

Checking out projects

You need to know the urls of the components you want to checkout. Then you would either of the following

# svn checkout url
# svn co url

The above will check it all out into a default folder name. For instance:

# svn checkout http://svn.your-server.org/your_repository/trunk

will check out your_repository into a folder called trunk in the current directory.

You can alternatively provide an optional folder name at the end of the command, to check the contents out into:

# svn checkout http://svn.your-server.org/your_repository/trunk foobar

This will check out your_repository into a folder called foobar.


SVN update and revert

An svn update merges the repository's changes with the local changed code, or shows places where conflicts have arisen (conflicts are shown with a C next to the files you tried to update). If someone else had committed changes to the repository and if these did not conflict with the changes in your own version, the svn update would show up a "G", not a "C". To deal with conflicts (C), see a later section.

# svn update <filename1> <filename2> ... <filenameN>
# svn revert <filename1> ... <filenameN>

You can update an entire directory (and subdirectories) by moving into that directory and typing:

> svn update


SVN diff

Note that svn diff does not connect to the repository! It compares your file against the file you had downloaded, not the file as it exists at the moment in the svn repository.

To find out what changes you've made to a greenstone source file:

# cd <folder containing source file>
# svn diff <filename>

e.g.

# svn diff file.cpp


Committing files

Perform an svn diff it to look over the changes made since you last updated the file.
Then svn update the file, perform the svn diff on the updated file, then svn commit it with a message:

# svn diff <filename>

Will show you the changes you've made since you last did an svn update on the file.

# svn update <filename>

It will merge changes made to the file in the repository with the changes you have on your machine.
Your own changes take precedence and these are preserved.
However, conflicts may arise if any of the lines that have been modified on your machine have been changed in the repository since you last checked it out/svn-updated it.

# svn diff <filename>

This diff now shows up the differences between your current file and what's there now in the repository.
Check that only the new code you have added is the sum total of the differences.

# svn commit - m "message" <filename>

Where the message string is one that explains what changes have been made, why you are committing the file or what purpose a new file serves.

To perform these SVN operations on more more than 1 file in one go, separate them by spaces:

# svn diff <filename> <filename2> <filenameN>
# svn update <filename> <filename2> <filenameN>
# svn diff <filename> <filename2> <filenameN>
# svn commit - m "message" <filename> <filename2> <filenameN>


Adding a new file to the repository

How to add a file (not for creating new folders in the svn repository):

# svn add <filename>
# svn commit -m "This new file will do something useful" <filename>

You can add more than one file at a time:

# svn add <filename1> <filename2> <filenameN>
# commit -m "These new files work together to add some extra functionality" <filename1> <filename2> <filenameN>


Deleting a file from the repository

To remove a file from the repository, you need to give it the url of the file in the repository:

# svn remove -m "I deleted this file for a reason" http://svn.your_server.org/....../thefile.ext</pre>

Aliases for svn remove are "svn delete" and "svn rm".
If more than one file needs to be removed, you need to perfom the svn remove operation for each file one after another. (You can't remove several files from the repository at the same time in one line.)


Helpful svn commands

# svn info
# svn switch
# svn status

If you do an "svn status" in a folder, it recursively lists all the Modified (M), Deleted (D) and Added (A) files.
It will also show up files that are in conflict (C) and those about which svn does not know (?).
To see what modifications were made to individual files marked with an M, you'd do an "svn diff":

# svn diff <filename>


# svn help
# svn help [item]
eg. svn help status

The last gives instructions on how to use an svn command. For example:

# svn help commit

Will tell you that to commit changes, you do "svn commit [path] --message "<reasoning>"


Conflicts and resolving them


If you performed an "svn update" on some files (or on a folder) and one or more came up with the status C then you have a conflict.
It means that changes to the same file on the same lines had been committed to the svn repository as what you have been working on.
The update could not successfully merge the corresponding lines as it didn't know which lines to keep and which to overwrite: the lines are in conflict.

If you ever encounter a file in conflict and you view it in an editor, you will see that conflicted lines will be marked with ===== and >>>>.
Both the changes you made and the conflicting changes in the repository will be embedded inside such special marks.
(If you had done an "svn update" on files that turned out to conflict, the action would have created a couple of additional versions of the file: conflict-filename.mine and conflict-filename.<revisionnumber>.
The first is your local version containing the changes you made. The second is the file as it is in the svn repository.)


To resolve conflicts in a file marked with a C

  1. Open up the file that's in conflict in an editor.
  2. Search for all occurrences of >>> or ===. Each occurrence marks a conflicted section and needs to be resolved.
  3. Deal with marked sections as appropriate: decide which parts you want to keep, which should be removed or how to combine (the best of) both.
  4. Once you've finished editing out the conflicts in the file, you have to set the file's status to resolved for it to be up to date (and updateable with svn):
# svn resolved <filename>

If you now try svn update on the file, it should no longer be marked as being in conflict.


Special SVN operations

svn log and annotate

# svn annotate <filename>

The above will list the code changes with NUMBERS in front of each line. For example,

# svn annotate GathererProg.java | less
# svn log <filename>

That will give all the messages for all the code changes. For example,

# svn log GathererProg.java | less
# svn log -rNUMBER <filename>

For example:

# svn log -r10242 GathererProg.java
# svn -v log http://svn.greenstone.org -r18201

Example output:

------------------------------------------------------------------------
r18201 | ak19 | 2008-12-15 14:10:06 +1300 (Mon, 15 Dec 2008) | 1 line
Changed paths:
  M /gsdl/trunk/perllib/unicode.pm
When associated files are renamed with the URLencoded versions of their original filenames,
the spaces are no longer URL encoded, 
as this conflicted with mp3, wmv and possibly other media file formats being opened in external 
or browser-embedded apps
------------------------------------------------------------------------


Creating a new project in the svn repository

You'd do this if you want to put your program, stored only on your machine at the moment, onto the svn repository.

The program folder on your harddrive, called "my_program" for instance, may contain:

If you want to put this project folder into the repository, inside http://svn.greenstone.org/other-projects/trunk/ then you would type the following in your x-term:

# cd my_program
# svn import -m "my message" . http://svn.greenstone.org/other-projects/trunk/my_program

That will put your folder and its contents into the svn repository inside a similarly named folder.
Now, we need to check out our own svn-managed copy:
Move up out of the local my_program directory and make a back-up copy of original program folder, just in case:

# cd .. 
# mv my_program my_program.bak

Finally, checkout the a copy of the program from the svn repository, that will be stored on your machine in my_program:

# svn co http://svn.greenstone.org/other-projects/trunk/my_program

Your program folder is just as before, except that it's now managed with svn.


Copying an older revision of a file or folder back into the current svn revision

If you've accidentally deleted an svn repository folder, such as "my_program", and want to bring it back (the older version is, after all, stored in the svn repository):

# svn copy -r <older version number> from-svn-url to-svn-url -m "message"

The revision version number you want to copy should be one where the folder (or file) you accidentally wiped out still exists. For example:

# svn copy -r 15315 http://svn.your_server.org/other-projects/trunk/my_program http://svn.your_server.org/other-projects/trunk/my_program -m "Accidental delete restored"

The above copies the folder of revision version 15315 at http://svn.your_server.org/other-projects/trunk/my_program into the same url at the current revision/head.


Checking out an older revision from SVN

In general, you would do:

/my/local/path/>svn update -r {2008-04-26}

If there were conflicts, delete everyting and checkout the older version:

/my/local/path/>svn co -r {2007-10-01} http://svn.your_server.org/your_repository/trunk .

Comparing versions: comparing current folder contents with contents of an older revision:

/my/local/path>svn diff -r {2008-04-26} .

Checking out an older revision of your_repository is a special situation. For this you would do:

# ant <target> -Dbranch.revision=<number>

eg.

# ant prepare -Dbranch.revision=15190

This will do an ant prepare/ant install/ant command using the revision number specified. Usually, you'd want to do an "ant prepare" with the revision flag. (And thereafter compile your your_repository up again with a normal "ant install".)


Changing an svn property

You may want to do this if a regular text file in the svn repository is marked as a binary file and therefore won't let you do an "svn diff" to compare the text contents.

An example situation:

/my/full/path>svn diff file.ext

Output:

Index: file.ext
___________________________________________________________________
Cannot display: file marked as a binary type.
svn:mime-type = application/octet-stream


To view a listing of the svn properties on this file:

/my/full/path>svn proplist file.ext

Output:

Properties on 'file.ext':
 svn:executable
 svn:mime-type


To edit the svn properties of this file:

/my/full/path>svn propedit svn:mime-type file.ext

Output:

svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR is set, and no 'editor-cmd' run-time configuration option was found
/my/full/path>export EDITOR=emacs

The above sets an editor to edit the svn properties with. In the example, it is the editor "emacs". On Windows you might set this to Notepad for instance.

Now you can choose to edit these properties:

/my/full/path>svn propedit svn:mime-type file.ext

If you only opened it up in an editor to have a good look at the contents but didn't make (or save) any changes, then when you close the editor, the output will likely be:

No changes to property 'svn:mime-type' on 'file.ext'

The troublesome property is the mime-type, which we delete as follows:

/my/full/path>svn propdel svn:mime-type file.ext

Output:

property 'svn:mime-type' deleted from 'file.ext'.

Now, if we do an "svn diff" on the file (which we couldn't do before because the file's MIME type was set to binary):

/my/full/path>svn diff file.ext

the output will be:

Property changes on: file.ext
___________________________________________________________________
Name: svn:mime-type
  - application/octet-stream
/my/full/path>svn commit -m "Property change from binary file into no longer having the mime-type property set (which didn't allow me to do a diff on the file)." file.ext
Sending        file.ext
Committed revision 16545.


That's all folks!

Marcin


<comments />

Ahmad said ...

<comment date="2012-04-17T01:19:58Z" name="Ahmad"> hisorry to ask, but i recog that u are key master at bespbrs.orgi made persian translation for version 1.0.1how can i officially introduce it and through svn?plz help </comment>

Basic Unix programming

                                    pdf_icon.png Download this article as a single PDF document 


Introduction

This is a guide to Unix programming. It is intended to be as generic as possible, since we want to create programs that are as portable as NetBSD itself.

Compiling a C program

I will assume you know C and how to edit a file under Unix.

First, type in the code to your file, let's say we put it in `hello.c'. Now, to create a program from it, do the following:

$ gcc hello.c -o hello
<compiler output>

If gcc (GNU Compiler Collection) is not available, use the command `cc' instead of `gcc'. Command line switches can be different, though. If you use C++, everything said here is the same, but you may replace the `gcc' by `g++'.

Now if there were no compiler errors, you can find the file `hello' in the current working directory. To execute this program, type

$ ./hello
<Your program's output>

The reason you can't just type `hello' is a good one but I won't explain it here. Beware, with some shells, the current line is overwritten when the program quits, so be sure to include at least a newline (\n) in the last line you print to the screen. This can be very frustrating if the program seems to give no output.

To compile larger programs, you can do this:

$ gcc -c file1.c
<compiler output>

And the same for file2.c etc. The -c switch tells gcc to only compile the code, and not link it into a complete program. This will result in a file with the same name as the source file, except the `.c' is replaced by `.o'.

Now, once you have compiled all your .c modules, you can tie the resulting object files (those with a .o extension) together with the following:

$ gcc file1.o file2.o file3.o -o program
<linker output>

Gcc now will link the object files together into your program. If all went good, you can run the program again with `./program'.

If you forget the -o switch on linking, the resulting default program will be called `a.out' and will be placed in the current working directory.

If an include file can't be found, you can use the -I switch to point gcc to the correct path. Example:

$ gcc -c -I/usr/local/include blah.c

This will compile blah.c into blah.o, and when it comes across a #include, it will look in /usr/local/include for that file if it can't find it in the default system include directory (or the current directory).

When you start using libraries (like libX11), you can use the -l, -L and -R flags when linking. This works as follows:

$ gcc -lX11 blah.o -o program

This will try to link the program to the file `libX11.so', or `libX11.a' depending on if your system is dynamically or statically linked.

Usually, the linker can't find libraries, so always use the following command:

$ gcc -L/usr/X11R6 -R/usr/X11R6 -lX11 blah.o -o program

The -L flag tells the linker where to find the `libX11.so' or `libX11.a' file. The -R flag is for `tying in' this path into the program. The reason this needs to be done next to the -L, is that when the system is dynamically linked, the library will be accessed on demand. On statically linked systems, the `libX11.a' file's contents will be copied into the final binary, which will make it a lot bigger. Nowadays, almost all systems are dynamically linked, so the `libX11.so' file is used. But the contents are not copied over. The library itself will be accessed whenever the program is started.

To make sure the dynamical linker can find the library, you need to tie in the library's path. Some systems (notably, most Linuxen) function even when you don't tie in the path yourself, because it `knows' about some common paths, but some other, more correct systems do not `know' any of the common paths. This seems to be for security reasons, but I don't know the exact details of why this is insecure (probably has to do with chrooted environments). It never hurts to use `-R', so please do so in every project you do, since you might some day want to run it on another system.

Using Make

It can be tedious work to type in those long command lines all the time, so you probably want to use Make to automate this task.

A Makefile generally looks like the following:

# This is a comment
program: hello.o
	gcc hello.o -o program

hello.o: hello.c
	gcc -c hello.c

Just save the things between the lines as the file called `Makefile' and type `make'. Make will try to build the first `target' (a word followed by a colon) it encounters, which is in our case `program'. It looks at what is required before it can do so, which is hello.o (after the colon). Warning: The whitespace in front of the commands below the target must be exactly one tab, or it will not work. This is a deeply propagated bug in all versions of Make. (don't let anyone convince you it's a feature. It's not)

So if `hello.o' does not exist, before Make can build `program', it must first build `hello.o'. This can be done from `hello.c', as it can see in the next target. If hello.o already existed, it will look if hello.c is newer (in date/time) than hello.o. If this is not the case, it will not have to rebuild hello.o, as obviously(?) the file hasn't been modified. If the file is newer, it will have to rebuild hello.o from hello.c.

When Make has determined that it must build a target, it will look at the indented lines (with a tab) directly following the target line. These are the command it will execute in sequence, expecting the target file to be a result of executing these commands.

If you wish to build a particular file only, just type `make <target>', where target is the name of the target you wish to build. Now we'll conclude with a bigger example of a Makefile:

# Assign variables.  Doing this makes the Makefile easier to
# modify if a path is incorrect, or another compiler is used.
CC=gcc
LINKFLAGS=-L/usr/X11R6/lib -R/usr/X11R6/lib

# If an `all' target is present, it is always executed by default
# (when you just type `make') even if it's not the first target.
all: myprog

# In the following, `${CC}' will expand to the contents of the
# variable `CC'.
myprog: first.o second.o
	${CC} ${LINKFLAGS} -lX11 first.o second.o  -o myprog

first.o: first.c
	${CC} -c first.c

second.o: second.c
	${CC} -c second.c

As you can see, you can do rather interesting things with Make.


Using BSD Make

Makefiles are nice, but typing the same lines all the time can get very annoying, even if you use SUFFIXES.

BSD's Make is a very nice Make, which comes pre-packed with some files which can make your life a lot easier and your Makefiles more elegant, but it is not compatible with GNU make. Some things are even incompatible between the Makes of different versions of BSD. Now we got that out of the way, let's see an example:

PROG=   test
SRCS=   test_a.c test_b.c
# We have written no manpage yet, so tell Make not to try and
# build it from nroff sources.  If you do have a manpage, you
#  usually won't need this line since the default name
#  of the manpage is ${PROG}.1 .
MAN=

.include <bsd.prog.mk>

That's all there's to it! Put this in the same directory as the `test' program's sources and you're good to go.

If you're on a non-BSD system, chances are that the normal `make' program will choke on this file. In that case, the BSD Make might be installed as `pmake', or `bmake'. On Mac OS X, BSD make is called `bsdmake', the default 'make' is GNU Make. If you can't find make on your particular system, ask your administrator about it.

The bsd.prog.mk file (in /usr/share/mk) does all the work of building the program, taking care of dependencies etc. This file also makes available a plethora of targets, like `all', `clean', `distclean' and even `install'. A good BSD Make implementation will even call `lint' on your source files to ensure your code is nice and clean.

If you wish to add flags to the C compiler, the clean way to do it is like this:

CFLAGS+= -I/usr/X11R6/include

For the linker, this is done by

LDADD=	-lX11

If you're adding libraries or include paths, be sure to make lint know about them:

LINTFLAGS+=	-lX11 -I/usr/X11R6/include

If you're creating a library, the Makefile looks slightly different:

LIB=    mylib
SRCS=   mylib_a.c mylib_b.c

.include <bsd.lib.mk>

A library doesn't have a manpage by default. You can force one to be built by supplying a MAN line, of course.

As you can see, the BSD Make system is extremely elegant for large projects. For simple projects also, but only if you have one program per directory. The system does not handle multiple programs in one directory at all. Of course, in large projects, using directories for each program is a must to keep the project structured, so this shouldn't be a major problem.

The main directory of a project should contain this Makefile:

SUBDIR=         test mylib

.include <bsd.subdir.mk>

Additionally, bsd.prog.mk and bsd.lib.mk always include the file ../Makefile.inc, so you can keep global settings (like DEBUG switches etc) in a Makefile.inc file at toplevel.

For more information, usually there is a /usr/share/mk/bsd.README file which explains BSD Make more completely than this little document. See also BSD Make.

Tip: Use LaTeX-mk if you want to build LaTeX sources with this type of BSD Makefiles.


Debugging programs with gdb

Now you know how to compile and link a program, but debugging is often very useful as well. I'll quickly explain some common things one can do with gdb, the GNU debugger.

At first, when you see someone using the debugger, it looks like yet another black art that can be done by gurus only, but it's not really too difficult, especially considering it's a command line debugger. If you practice a bit, using gdb will become a second nature.

First, it is handy to compile a program with debugging symbols. This means the debugger knows about the variable and function names you use in your program. To do this, use the -g switch for gcc when compiling:

$ gcc -g blah.c -o program

For each object, you must use -g on compilation. On linking -g can beomitted. Now, run the debugger:

$ gdb program
<output and prompt>

To run the program, just type `run'. To load a new binary into gdb, use `file <binary>'. For information about possible commands, use `help'.

When your program crashes, you can use the command `bt' to examine the stack. With `select <N>' you can select stack frame N. For example, suppose you're in the middle of a function foo which is called from function bar, you can switch to bar by going down a step in the stack. Stack frame 0 is always the last called function.

With `l' (or `list') you can list a certain file, at a certain point. By default `l' shows the point where the debugger has halted right now. It always lists 10 lines of context, with the middle line being the current line. If you wish to see some other file or some other line, use `l file.c:line', with file.c being the file to look at, line the number of the line. With every consecutive `l', the next lines will be shown, so you can scroll through the program by hitting `l' all the time.

The nice thing about gdb is, that when you simply press enter, it executes the last command. So you need to type `l' only once, and keep pressing enter until you see the right piece of code. This is especially useful when using `step' (see below).

To investigate a variable, just use `print <varname>'. You can print contents of pointers in the same fashion as you would in C itself, by prefixing it with a *. You can also cast to int, char etc if you wish to see only that many bits of the memory (useful when investigating buffers). Printing a variable only works if you've selected a stack frame in which the variable has a meaning. So if function foo has an integer called 'i' and function bar also has one, it depends on the selected stack frame which of the two i's contents you get to see.

With `break' you can set breakpoints. The program being debugged will stop when it gets to the line of the breakpoint. Breakpoints work like listings when giving line numbers/filenames. You can clear a breakpoint by issuing the same, but with `break' replaced by `clear'.

To continue the program, use `continue'. This just resumes executing the program from the point where it halted. If the program really crashed, continue is of course not possible.

To walk through a program line by line, use `step'. This way, after each line is executed, you can investigate how variables changed. It is common practice to set a breakpoint at a place in the program where you expect a problem to occur. When that breakpoint is hit, you can use `step' to step through the failing part of the program.

If you think a certain function will work, use `finish', which is like a continue, but only for the current function. When the previous function is recalled from the stack, the program will stop.

With `kill' you can immediately kill your program. `quit' quits gdb.

References


Marcin


<comments />

Cindy said ...

<comment date="2012-07-11T22:52:13Z" name="Cindy"> Daniel'i had my own, it worked for my bios just not hdd, your tlakin bout making money with it wich doesnt sound cool, its out there free for people and if people ended up on this page then they are probly lookin for a free solution , and tada . here it is, if you wanna go and make money from free programs people can find online then feel free to but why bother bitchn about it here why not go and advertise you services somewhere else or put on a wig, dress and lipstick and bendover in a backally somewhere for a few bucks. doth for get to swallow. </comment>

GCC warnings explained

                                    pdf_icon.png Download this article as a single PDF document 


GCC Compiler Errors and How-to Solve The Problem

Introduction

Sometimes GCC emits something that can be described as Haiku poems - and you have no clue as to what it really is complaining about. This page is a collection of such gems, their meaning in English and how to solve the problem.

Compilation Errors

This is a list of compilation errors that you might find yourself trying to interpret in no particular order.


Error: ...discards qualifiers

Error message: passing ?const ClassName? as ?this? argument of ?virtual void ClassName::methodName()? discards qualifiers.

You have called a method that isn't const using a const object (const ClassName *foo). Either add const to you method, e.g.

class ClassName
{
public:
  void methodName() const;
};

Alternatively, you remove the const from your object, declaring it as ClassName *foo instead of const ClassName *foo.
Sometimes it is possible to solve this issue using const_cast .


Error: storage size of 'foo' isn't known

Error message: storage size of 'foo' isn't known.
I ran into this problem when using a typedefed struct as a struct from a function.

typedef struct { ... } Foo;

void function()
{
 struct Foo foo;

 ...
}

The solution is trivial - but hard to spot if you don't know what to look for.

typedef struct { ... } Foo;

void function()
{
  struct Foo foo;

 ...
}


Error: multiple types in one declaration

Error message: multiple types in one declaration.

You've probably forgot to end a class declaration with a semi-colon - ;.
The faulting class is not the one that GCC complains about but one of the classes included from the file containing the class declaration GCC nags you about.
Locate the faulty class in one of the suspected files, add the semi-colon and try compiling again.


Error: invalid use of undefined type ?struct Foo?

Error message: invalid use of undefined type ?struct Foo?.

It is likely that you are trying to use the class Foo that you've forward declared but never included.
Simply include the full class declaration for Foo and everything will work.


Error: no matching function for call to 'FooClass::foo()'

Error message: no matching function for call to 'FooClass::foo()'.

If you have implemented and declared the member foo, you are probably trying to use a method from a forward declared class. You need to include the header file containing the declaration of FooClass.

Another variant of this is when you are missing an inherited method. As you are using a forward declared type, GCC cannot tell if FooClass inherits the class implementing foo.
A concrete example of this:

chatmaster.cpp:220: error: no matching function for call to ?ChatMaster::connect(const ContactList*&, const char [35], ChatMaster* const, const char [39])?
/usr/lib/qt3/include/qobject.h:116: note: candidates are: static bool QObject::connect(const QObject*, const char*, const QObject*, const char*)
/usr/lib/qt3/include/qobject.h:226: note: bool QObject::connect(const QObject*, const char*, const char*) const


Here ContactList is forward declared and GCC cannot tell that it inherits QObject (containing the connect method). You will not get any message telling you that you have forgot to include the header file.


Error: undefined reference to 'FooClass::foo()'

Error message: undefined reference to 'FooClass::foo()' .

You have defined foo in the header file, but not implemented it. Alternately, you've used a library function without linking to the needed library.


Error: invalid operands of types `const char[31]' and `const char[7]' to binary `operator+'

Error message: invalid operands of types `const char[31]' and `const char[7]' to binary `operator+'.

You cannot write "foo" + "bar" , instead write "foo" "bar" (the split could be across a line-break). GCC automatically concatenates strings.


Error: `QValueList' undeclared (first use this function)

Error message: `QValueList' undeclared (first use this function).

This happens when you write QValueList foo instead of QValueList<type> foo.


Error: cannot call member function `Foo* Foo::instance() const' without object

Error message: cannot call member function `Foo* Foo::instance() const' without object.

You have called instance as a static method, but is was not declared as such.


Errors: non-pointer type, non-aggregate type, cannot convert

Error message: base operand of `->' has non-pointer type `Foo'.
Error message: request for member `bar' in `foo', which is of non-aggregate type `Foo*' .
Error message: cannot convert `Foo' to `Foo*' in initialization.

These are all examples of messages that you run into when mixing references, pointers and stack-based variables.


Error: syntax error before `*' token

Error message: syntax error before `*' token.

A class name is unknown. It has not been forward declared, nor included.


Error: `foo' is not a type

Error message: `foo' is not a type.

You wrote foo.width() when you meant foo->width().


Error: unable to find a register to spill in class `FOO'

Error message: unable to find a register to spill in class `FOO'

This error message isn't telling you about an error in your code, it's telling you about an internal failure (almost certainly a bug) in the compiler.

You might be able to work around the compiler bug by re-working your code, but it's not at all obvious how.
You might also try tweaking command-line options, particularly ones related to optimization.
A Google search for the error message might be fruitful.

Simon Farnsworth also points out that the cause of this can be inline assembler code: This error can also be caused by inline assembly code.
If you have asm() directives in your source (see http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.htm ), check that the input and output operand constraints are correct, and consider relaxing constraints where possible.
However, be aware that your inline assembly may not run as fast as intended if you do this.


Error: invalid operands to binary ?operator<<?

Error message: invalid operands of types ?<unresolved overloaded function type>? and ?const char [15]? to binary ?operator<<? .

This message was produced by misspelling cout, for instance:

count << "Hello world!" << endl;

Apparently, there is a std::count function that confuses the compiler to produce this rather cryptic error message.


Qt Peculiarities

Sometimes Qt's build system and GCC steps on one anothers toes, resulting in confusion.
Many issues can be solved through a make clean && make or just a touch myproject.pro && make.


Using Qt - invalid use of void expression

Error message: invalid use of void expression, while using Qt.

This message can appear if you have forgotten a SIGNAL() or SLOT() macro when calling QObject::connect or a similar function.


Using Qt - ...before ?protected?

Error message: 'refers to protected, but at an odd line number, while using Qt.

The signals keyword is really just a define of protected. All signals are protected methods. Hence, the compiler can refer to your signals section when it mentions protected.


Using Qt - ...vtable

Error message: complaints about vtable entries.

This can indicate that you are missing a Q_OBJECT macro, or have missed to implement a virtual method. Check for this.

Another potential cause can be that QMake generates Makefiles that can run into this issue when your add or remove signals and slots to classes.
The first thing to try is to touch your pro-file, e.g. run touch foo.pro from the command line.
If you cannot find an actual problem, and touch does not work, try running make distclean && qmake && make to do a clean rebuild.


Serious Warnings

The warning messages listed below indicates that you might run into serious trouble. As we're talking about warnings, you will encounter and have to trac down these issues at run-time. A better option is to do something about the warnings.

If you want warnings to stop your compilation, run GCC with the flag -Werror.
Also, see the relevant GCC documentation.


Warning: Control reaches the end of a non-void function

Warning message: Control reaches the end of a non-void function.

There is a way to reach the end of a non-void function without returning something.
Add a final return at the end of the function to solve this.

Do not ignore this warning - it is possible to run into really hard to debug problems if you do.


Warning: ?foo? is used uninitialized in this function

Warning message: ?foo? is used uninitialized in this function .

You are using foo, even though you have not initialized it. This listing will cause the problem:

int main()
{
  int foo, bar;

  bar = foo;
  foo = 7;

  return bar;
}

The solution is to initialize foo before using it. For example, add foo = 0; before bar = foo;.
Ignoring this warning can give you random values in a variable, causing the potential bug to appear sometimes, but not always. This can make it very hard to debug.


Warning: cannot pass objects of non-POD type 'struct std::string' through '...'

Warning message: cannot pass objects of non-POD type 'struct std::string' through '...'; call will abort at runtime.

You are, for instance, trying to print a C++ std::string directly to printf.

std::string foo;
printf( "Foo: %s\n", foo );

The result in run-time can be something like Illegal instruction (core dumped).
The proper way to handle the std::string to printf is to use the c_str method:

std::string foo;
printf( "Foo: %s\n", foo.c_str() );


Additional sources


Marcin


<comments />