Knowledge Archives
http://www.co-internet.net/net/articles/mythmanualsub.txt.html

The Myth of Manual Search Engine Submission
------------------------------------------------------
by
Ralph Tegtmeier
------------------------------------------------------

One of the ongoing myths abounding in the search
engine world is that of manual submission: many search
engine optimization agencies make a point in promoting
their "manual submission services", and if you follow
discussions in various forums and newsgroups you will
frequently encounter people advising others to "submit
manually only, or else ..."

Well - or else, what? Is there something inherently
evil or - at the very least - inefficient about
automatic submission? And if so, where's the problem?
To cut a long story short: there really isn't any, but
to understand why not, it's important to get a grasp
on the actual mechanics of search engine submission.
This will help debunk a lot of the mystique currently
surrounding the process and may save you hundreds of
hours of unnecessary manual labor.

All important search engines feature special
submission pages endowed with HTML forms where you can
enter your web site's URL or, in some cases,
additional data such as your email address, your name,
etc. Submission is effected by clicking on the submit
button, whereupon you will typically be directed to a
confirmation page. To avoid spam and oversubmissions,
URLs will only be accepted if and when the data is
referred by these same submission pages. Of course,
when submitting manually this is a given, and the user
needn't even be aware of it.

The trick with automatic submission is to make
believe, i.e. to fool the search engine submission
setup that the URL submitted was actually entered on
the submission page installed for this purpose. So how
is this done? Easy: every web browser carries a
variable with it which can be read by any server
accessed. In technical terms: it is the "http_referer"
(yes, only one "r"!) variable which tells the search
engine from where the browser accessing the success or
confirmation page is actually coming. If you submit a
site manually, your browser will automatically convey
this variable to the submission program (which, by the
way, is typically a Perl or PHP script). So what every
automatic submission tool has to do to see the
submitted data accepted, is to pose as a web browser
by emulating a browsers identification variable (the
"UserAgent"), configure its http_referer variable to
the search engine's submission page's - and bingo!

If this is done right, and all commercial submission
programs we are aware of are intelligent enough to
cover this very basic routine, there's really no way a
search engine can tell whether any given URL was
submitted by a robot program, i.e. automatically, or
manually.

Of course, there are some telltale hints which may yet
either give your automatic submitter away or provoke
a submission denial:
* If your program submits too many pages in too short
an interval
* If your submitter program fails to convey other
required data beyond the mere URL (e.g. your email
address, your name, or whatever is required by the
search engine), as any diligent human submitter
would
* If your program is not up-to-date, referencing an
obsolete submission page variable (these tend to
change from time to time as search engines develop)

Again, all commercial programs we know of, whether
client or server based, will do all this quite nicely.
Just make sure you are using a current version. To
avoid infringement of submission maximums, submitting
no more than 5 pages per domain and engine is
generally considered a safe approach. If submitting
multiple URLs to several engines (as most programs
will do), make sure these are submitted to each engine
consecutively: first, you would, for example, submit
URL #1 to AltaVista, next to Excite, then to FAST,
then to Google, next to HotBot, next to Northern
Light, and so on, till the very last engine on your
list. Only after this list of engines has been
submitted to, would you (or, rather, your program)
start submitting URL #2, etc. This ensures that you
won't run afault of submission time limits like you
might if you submitted all URLs first to AltaVista in
one go, then to Excite, etc.

Recommendations
---------------
One of the most powerful and reliable client based
submission tools (and certainly the best money value)
is DumpTruck. We are very familiar with it because we
have used it extensively before developing our own
server based solution, fantomas subFrog(TM) (for which
see below) - and because we (in our previous
incarnation as "tsigaan softer systems") wrote the
code for the German engines submission routine
ourselves.
Read more about it here:
< http://fantomaster.com/fasmbres02.html#dumptruck >

For a fully professional server based solution
allowing for bulk submissions as a system background
process not requiring any monitoring and offering
duplicate email report generation (e.g. one copy to
yourself, one to your clients) see our fantomas
subFrog(TM). In contrast to most submission software
available on the net, our program caters not only to
US based search engines but covers many international
engines as well. Moreover, if you want to see more
engines added, we will upgrade the program at no extra
charge, provided their submission structure allows for
automatization at all.
Check it out here:
< http://fantomaster.com/fasubfrog0.html >

This text may freely be republished or distributed in unmodified form provided the following resource box is included intact either at the beginning or the end of the article and a complimentary copy or notice (link) is sent to the author at the address specified below:

Ralph Tegtmeier is the co-founder and principal of fantomaster.com GmbH (Belgium), < http://fantomaster.com/ >, a company specializing in webmasters software development, industrial-strength cloaking and search engine positioning services.

He has been a web marketer since 1994 and is editor-in-chief of fantomNews, a free newsletter focusing on search engine optimization, available at: < http://fantomaster.com/fantomnews-sub.html > You can contact him at mailto:fneditor@fantomaster.com
(c) copyright 2002 by fantomaster.com
All rights reserved.
Downloaded at: < http://fantomaster.com/ >