=head1 NAME

INSTALL - Swish-e Installation Instructions

=head1 OVERVIEW

This document describes how to download, build and install Swish-e from
source.  Also below is a basic overview of using Swish-e to index documents
with pointers to other more advanced examples.

This document also provides instructions on how to get help installing and
using Swish-e (and the important information you should provide when asking
for help). Please read these instructions before requesting help on the
Swish-e discussion list.  
See L<"QUESTIONS AND TROUBLESHOOTING">.

Although building from source is recommended, some OS distributions (e.g.
Debian) provide pre-compiled binaries.  Check with your distribution for
available packages. Build from source if your distribution does not offer
the current version of swish-e.

A Windows binary version is available as a separate download from the
Swish-e site (http://swish-e.org).  Many of the installation instructions
below will not apply to Windows users. One exception is that the Perl LWP
library is required for spidering remote sites.

Also, please read the Swish-e FAQ L<SWISH-FAQ|SWISH-FAQ> as it answers
many frequently asked questions.

Swish-e knows how to index HTML, XML, and plain text documents.  Helper applications
and other tools are used to convert documents such as PDF or MS Word into a format
that swish-e can index.  These additional applications and tools (listed below) must be installed
separately.

NOTE: Swish-e version 4.2.0 installs a lot more files when running "make install".  Be aware 
that the Swish-e documentation may thus include errors about where files are located.

=head1 SYSTEM REQUIREMENTS

Swish-e makes use of a number of libraries and tools that are not distributed with Swish-e.
Some libraries need to be installed before building Swish-e from source, where other tools
can be installed at any time.  See below for details.

=head2 Software Requirements

Swish-e is written in C and up to this time has been tested on a number of
platforms, including Sun/Solaris, Dec Alpha, BSD, Linux, OS X, and Open VMS.

The GNU C compiler, gcc, and GNU make are strongly recommended.

=head2 Optional but Recommended Packages

Most of these packages are available as easily installable packages.  Check your
operating system vendor, or install from source.  Most are very common 
packages that may already be installed on your computer.

As noted below, some packages need to be installed before building Swish-e
from source, while others may be added after Swish-e is installed. 

=over 4

=item * Libxml2

Libxml2 is very strongly recommended.  It is used for parsing both HTML and
XML files. Swish-e can be built and installed without libxml2, but the HTML
parser built into swish-e is not as accurate as libxml2.

    http://xmlsoft.org/

For swish-e to use libxml2 it must be installed before building swish-e.

=item * Zlib Compression

The Zlib compression library is commonly installed on most systems and is
recommended for use with Swish-e.  Zlib is used for compressing text stored
in the swish-e index.

    http://www.gzip.org/zlib/

Zlib must be installed before building swish-e.

=item * LWP Perl Modules

The LWP bundle of Perl modules is used for spidering web servers.  If you
plan to index documents located on a web server you must have the LWP bundle
of modules installed. 

Most Perl installations include the LWP Perl modules.  Type "perldoc LWP" to
see if LWP is installed on your system.  If not, contact your system
administrator or visit http://search.cpan.org to locate the LWP bundle:



Windows users can use the "ppm" command to install the LWP library (although
should be installed automatically with current versions of Perl).

=item * MIME::Types

This Perl module is not requires, but if installed can be helpful in filtering
(converting) files based on their content-type.  MIME::Types is used by
the SWISH::Filter module.

=item * Indexing PDF Documents

Indexing PDF files requires the xpdf package.  This is a common package
available with most operating systems and often provided as a package.

    http://www.foolabs.com/xpdf/

Xpdf may be added after swish-e is installed.

=item * Indexing MS Word Documents

Indexing MS Word documents requires the Catdoc program.

    http://www.45.free.net/~vitus/ice/catdoc

Catdoc may be added after swish-e is installed

=item * Indexing MP3 ID3 Tags

Indexing MP3 ID3 Tags requires the MP3::Tag Perl module.  See
http://search.cpan.org. MP3::Tag may be installed after swish-e is
installed.

=item * Indexing MS Excel Files

Indexing MS Excel files is supported by the following Perl modules, also available at
http://search.cpan.org.

    Spreadsheet::ParseExcel
    HTML::Entities

These Perl modules may be installed after swish-e is installed.

=back

=head1 INSTALLATION

Here are brief installation instructions that should work in most cases.
Following this section are more detailed instructions and examples.

=head2 Building Swish-e

Download swish-e using your favorite web browser or a utility like wget, lynx, or
lwp-download.  Unpack and build using the following steps:

Note: "swish-e-2.4.0" is used as an example.  Download the most current version
available and adjust the commands below!  Also, if running Debian, see notes below on
building a .deb package.

The "$" symbol indicates steps run as an unprivileged user.  The "#" 
indicates steps run as the superuser (root).


    $ wget http://swish-e.org/Download/swish-e-2.4.0.tar.gz
    $ gzip -dc swihs-e-2.4.0.tar.gz | tar xof -
    $ cd swish-e-2.4.0  (this directory will depend on the version of Swish-e)

    $ ./configure
    $ make
    $ make check
    ...
    ==================
    All 3 tests passed
    ==================

    $ su root  (or use sudo)
    (enter password)

    # make install
    # exit
    $ swish-e -V
    SWISH-E 2.4.0

B<IMPORTANT:> Once installed do not run swish-e as the superuser (root) --
root is only required during the installation step when installing into
system directories.

Here's another installation example.  This might be used if you do not have
root access or you wish to install swish someplace other than /usr/local.

This example also shows building Swish-e in a "build" directory separate 
from where the source files.  This is the recommended way to build Swish-e,
but requires GNU Make.  Without GNU Make you will likely need to build
from within the source directory as shown in the previous example.

    $ tar zxof swish-e-2.4.0.tar.gz  (GNU tar with "z" option)
    $ mkdir build 
    $ cd build

Note that the current directory is not where Swish-e was unpacked.

Swish-e uses a F<configure> script.  F<configure> has many options, but reasonable
and standard defaults.  Running 

    $ ../swish-e-2.4.0/configure --help

will display the options.  Two options are of common interest: --prefix sets
the top-level installation directory, and --disable-shared will link swish-e
statically, which may be needed on some platforms (Solaris 2.6 perhaps).

Now configure and build Swish-e:

    $ ../swish-e-2.4.0/configure --prefix=$HOME/swish-e
    $ make >/dev/null  (redirect output to only see warnings and errors)
    $ make check
    ...
    ==================
    All 3 tests passed
    ==================

    $ make install
    $ $HOME/swish-e/bin/swish-e -V 
    SWISH-E 2.4.0

In this case you would likely want to add $HOME/swish-e/bin to your shell's 
PATH setting.

=head2 Building a Debian Package

The Swish-e distribution includes the files requires to build a Debian pakcage.

    $ tar zxof swish-e-2.4.0.tar.gz  (GNU tar with "z" option)
    $ cd swish-e-2.4.0
    $ fakeroot debian/rules binary
    [lots of output]
    dpkg-deb: building package `swish-e' in `../swish-e_2.4.0-0_i386.deb'.
    $ su
    # dpkg -i ../swish-e_2.4.0-0_i386.deb


=head2 What's installed

Swish installs a number of files.  By default all files are installed below
/usr/local, but this can be changed by setting --prefix when running
F<configure> (as shown above).  Individual paths may also be set.  Run
F<configure --help> for details.


   $prefix/bin/swish-e         The swish-e binary program
   $prefix/share/doc/swish-e/  Full documentation and examples
   $prefix/lib/libswish-e      The swish-e C library
   $prefix/include/swish-e.h   The library header file
   $prefix/man/man1/           Documentation as manual pages
   $prefix/lib/swish-e/        Helper programs (spider.pl, swishspider, swish.cgi)
   $prefix/lib/swish-e/perl/   Perl helper modules


Note that the Perl modules are I<not> installed in the system Perl library.
Swish-e and the Perl scripts that require the modules know where to find the
modules, but the F<perldoc> program used for reading documentation does not. 
This can be corrected by adding $prefix/lib/swish-e and
$prefix/lib/swish-e/perl to the PERL5LIB environment variable.

=head2 Documentation

Documentation can be found in the $prefix/share/doc/swish-e directory.
Documentation can also be read on-line at the Swish-e web site:

    http://swish-e.org.

=head2 The Swish-e documentation as man(1) pages

Running "make install" installs some of the Swish-e documentation as man pages.
The following man pages are installed:

    SWISH-FAQ(1)
    SWISH-CONFIG(1)
    SWISH-RUN(1)
    SWISH-LIBRARY(1)

The man pages are installed in the system man directory.  This directory
is determined by running ./configure and can be set by passing the
directory when running ./configure.

For example,

    ./configure --mandir=/usr/local/doc/man

Information on running ./configure can be found by typing:

    ./configure --help

=head2 Join the Swish-e discussion list

The final step when installing Swish-e is to join the Swish-e discussion list.

The Swish-e discussion list is the place to ask questions about installing and using
Swish-e, see or post bug fixes or security announcements, and a place where B<you> can offer
help to others.  Please do not contact the developers directly.

The list is typically I<very low traffic>, so it won't overload your inbox.  Please take
time to subscribe.  See http://Swish-e.org.

If you are using Swish-e on a public site, please let the list know so it can be added to
the list of sites that use Swish-e!

Please review the next section before posting a question to the Swish-e list.

=head1 QUESTIONS AND TROUBLESHOOTING

Support for installation, configuration and usage is available via the
Swish-e discussion list.  Visit http://swish-e.org for information.
Do not contact developers directly for help -- always post your question
to the list.

It's very important to provide the right information when asking
for help.  

Please search the Swish-e list archive before posting a question, and
check the L<SWISH-FAQ|SWISH-FAQ> to see if your question hasn't already
been asked.  

Before posting use tools available to narrow down the problem.

Swish-e has the -T, -v, and -k switches that may help resolve issues. These
switches are described on the L<SWISH-RUN|SWISH-RUN> page. For example, if
you cannot find a document by a keyword that you believe should be indexed
try indexing just that single file, and use the -T INDEXED_WORDS option to
see if the word is actually being indexed.  First try without any changes to
default settings:

    swish-e -i testdoc.html -T indexed_words | less

if that works then add in your configuration file:

    swish-e -i testdoc.html -c swish.conf -T indexed_words | less

If that still isn't working as you expect try to reduce the test document to
a very small example.  This will be very helpful when asking for help.

Other tools are to use -H9 when searching to display full headers in search
results.  Look at the "Parsed Words" header to see what words swish-e is
searching for.

=head2 When posting please provide the following information:

=over 4

=item *

The exact version of Swish-e that you are using.  Running Swish-e with the
C<-V> switch will print the version number.  Also, supply the output from
C<uname -a> or similar command that identifies the operating system you
are running on.  If you are running an old version of swish be prepared
for a response to your question of "upgrade."

=item *

A summary of the problem.  This should include the commands issued
(e.g. for indexing or searching) and their output, and why you don't
think it's working correctly.  Please cut-n-paste the exact commands
and their output instead of retyping to avoid errors.

=item *

Include a copy of the configuration file you are using, if any.  Swish-e
has reasonable defaults so in many cases you can run it without using
a configuration file.  But, if you need to use a configuration file,
B<reduce it down> to the absolute minimum number of commands required to
demonstrate your problem.  Again, cut-n-paste.

=item *

A small copy of a source document that demonstrates the problem.

If you are having problems spidering a web server, use lwp-download or
wget to copy the file locally to make sure you can index the document
using the file system method.  This will help determine if the problem is
with spidering or with indexing.

If you expect help with spidering, don't post fake URLs, as it makes it
impossible to test.  If you don't want to expose your web page to the
people on the Swish-e list, find some other site to test spidering on.
If that works, but you still cannot spider your own site then post your
real URL if you want help, or make a test document available via some other 
source.

=item *

If you are having trouble building Swish-e please cut-n-paste the output
from make (or from ./configure if that's where the problem is).

=back

The key is to provide enough information so that others may reproduce the problem. 

=head1 ADDITIONAL INSTALLATION OPTIONS

These steps are not required for normal use of Swish-e.

=head2 The SWISH::API Perl Module

The Swish-e distribution includes a module that provides a Perl
interface to the Swish-e C library.  This module provides a way
to search a Swish-e index without running the swish-e program.
Searching an index will be many times faster when running under a
persistent environment such as Apache/mod_perl with the SWISH::API
module.

See the F<perl/README> file for information on installing and
using the SWISH::API Perl module.


=head2 Creating PDF and Postscript documentation

The Swish-e documentation in HTML format was created with Pod::HtmlPsPdf,
a package of Perl modules written and/or modified by Stas Bekman to automate
the conversion of documents in pod format (see perldoc perlpod) to HTML,
Postscript, and PDF.  A slightly modified version of this package is
include with the Swish-e distribution and used for building the HTML.

If your system has the B<necessary tools> to build Postscript and the
converter ps2pdf installed, you may be able to build the Postscript
and PDF versions of the documentation.  After you have run ./configure,
type from the F<doc> directory of the distribution:

    make pdf

And with any luck you will end up with the these two files in the top-level directory:

    swish-e_documentation.pdf
    swish-e_documentation.ps

Most people find reading the documentation in HTML most convenient.

=head1 GENERAL CONFIGURATION AND USAGE

This section should give you a basic overview of indexing and searching
with B<Swish-e>.  Other examples can be found in the F<conf> directory which will
step you through a number of different configurations.
Also, please review the L<SWISH-FAQ|SWISH-FAQ>.

=head2 Introduction to Indexing and Searching

Swish-e can index files on the local file system.  For example, running:

     swish-e -i /var/www/htdocs

will index I<all> files in the /var/www/htdocs directory.  You may specify
one or more files or directories with the -i option.  By default this will
create an index (which is made up of more than one file) in the current
directory called F<index.swish-e>.

Then to search the resulting index for a given word:

     swish-e -w apache

This will find the word "apache" in the body or title of the indexed 
documents.

As mentioned above, Swish-e will index all files in a directory unless
instructed otherwise. So if /var/www/htdocs contains non-HTML then you will
need a configuration file to limit the files that Swish-e indexes.  Create a
file called "swish.conf":

    # Example configuration file

    # Tell swish what to index (same as -i switch above)
    IndexDir /var/www/htdocs

    # Only index HTML and text files
    IndexOnly .htm .html .txt

    # Tell swish that .txt files are to use the text parser.
    IndexContents TXT* .txt

    # Otherwise, use the HTML parser
    DefaultContents HTML*

Save that as "swish.conf" and reindex:

    swish-e -c swish.conf

The Swish-e configuration settings are described in the
L<SWISH-CONFIG|SWISH-CONFIG> manual page.  Order of statements in the
configuration file is typically not important, although some statements
depend on previously set statements.  There are many possible settings. 
Good advice is to use as few settings as possible when first starting out
with Swish-e.

The runtime options (switches) are described in the L<SWISH-RUN|SWISH-RUN>
manual page.  You may also see a summary of options by running:

    swish-e -h

Swish-e has two other methods reading input files.  One method uses a Perl helper script and 
the LWP Perl library to spider remote web sites:

    swish-e -S http -i http://localhost/index.html -v2

This will spider the web server running on the local host.  The C<-S> option defines the
input source method to be "http", C<-i> specifies the URL to spider, and C<-v> sets the
verbose level to two.  There are a number of configuration options specific to the C<-S>
http input source.  See L<SWISH-CONFIG|SWISH-CONFIG>.  Note that only files of Content-Type
text/* will be indexed.

The C<-S http> method is depreciated in favor of the next input method.

The other method is a general purpose input method where Swish-e reads input from a program 
that produces documents in a special format.  The program might read and format data stored 
in a database, or parse and format messages in a mailing list archive, or run a program that 
spiders web sites like the previous method.

The Swish-e distribution includes a spider program that uses this method of
input.  This spider program is much more configurable and feature-rich than
the previous -S http method.

To duplicate the previous example create a configuration file called
"swish2.conf"

    # Example for spidering
    # Use the "spider.pl" program included with Swish-e
    IndexDir spider.pl

    # Define what site to index
    SwishProgParameters default http://localhost/index.html

Then create the index using the command:

    swish-e -S prog -c swish2.conf

This says to use the C<-S prog> input source method.  Note that in this case
the IndexDir settings does not list a file or directory to index, but a
program name run.  This program, spider.pl, does the work of fetching the
documents from the web server and passing them to Swish-e for indexing.

The SwishProgParameters options is a special feature that allows passing command line 
parameters to the program specified with IndexDir.  In this case passing the word "default" 
which tells spider.pl to use default settings, and the URL to spider.

Running a script under Windows requires specifying the interpreter (e.g.
perl.exe) and then use SwishPropParameters to specify the script and the
script's parameters. See I<Notes when using -S prog on MS Windows> on the
L<SWISH-RUN|SWISH-RUN> page.

The advantage of the C<-S prog> method of spidering (over the previous C<-S http> method) is
that the Perl code is only compiled once instead of for every document fetched from the web
server.  In addition it is a much more advanced spider with many, many features.  Still, as
used here, spider.pl will automatically index PDF or MS Word documents if (when) Xpdf and
Catdoc are installed.

A special form of the C<-S prog> input source method is:

    ./myprog --option | swish-e -S prog -i stdin -c config

This allows running Swish-e from a program (instead of running the external program from 
Swish-e).  Thus, this also can be done:

    ./myprog --option > outfile
    swish-e -S prog -i stdin -c config < outfile

or

    ./myprog --option > outfile
    cat outfile | swish-e -S prog -i stdin -c config

One final note about the C<-S prog> input source method.  The program specified with -i or
IndexDir needs to be an absolute path.  The exception is when the program is
installed in the "libexecdir" directory and then a plain program name may be 
specified (as in the example showing spider.pl above).

All three input source methods are described in more detail on the  L<SWISH-RUN|SWISH-RUN> page.

=head2 Metanames and Properties

There's two key Swish-e concepts that you need to be familiar with: 
Metanames and Properties.

=over 4

=item * Metanames

Swish-e creates a reverse index.  Just like an index in a book, you look up
a word and it lists the pages (or documents) where that word can be found.

Swish-e can create multiple index tables within the same index file.  For
example, you might want to create an index of just words in HTML titles so
searches can be limited to just titles.  Or you might have descriptive 
words in a meta tag called "keywords" you would like to search.

Some database systems might call these different "fields" or "columns", but
swish-e calls them I<MetaNames> (as a result of first indexing HTML meta
tags).

To find documents with "foo" in their title you might run:

    swish-e -w swishtitle=foo

or more advanced:

    swish-e -w swishtitle=(foo or bar) or swishdefault=(baz)

The Metaname "swishdefault" is the name used by Swish-e if no other name is
specified. The following two searches are the same:

    swish-e -w foo
    swish-e -w swishdefault=foo

When indexing HTML documents Swish-e indexes words in the body and title 
under the Metaname "swishdefault".

=item * Properties

Swish-e search results is a list of files -- actually internally swish uses
file numbers.  Data can be associated with each file number when indexing. 
For example, by default Swish-e associates the file's name, title, last
modified date, and size with the file number and these items can be printed
in search results.  In Swish-e this associated data is called a file's
I<Properties>.  Properties can be any data you wish to associated with a
document -- even the entire text of the document can be stored in the index. 
What data is stored as a Property is controlled by the I<PropertyNames> (and
others) configuration directive.

What properties are printed with search results depends on the -x or -p switches.  By default
Swish-e returns the rank, path/URL, title and file size in bytes for each result.

=back

=head2 Getting Started With Swish-e

Swish-e reads a configuration file (see L<SWISH-CONFIG|SWISH-CONFIG>)
for directives that control what and how Swish-e indexes files.
Swish-e is also controlled by command line arguments (see
L<SWISH-RUN|SWISH-RUN>).  Many of the command line arguments have equivalent 
configuration directives (e.g. -i and IndexDir).

Swish-e does not require a configuration file, but
most people need to change the default behavior by placing settings
in a configuration file.

To try the examples below you may change to the F<tests> subdirectory of the
distribution.  The tests will use the *.html files in this directory when
creating the test index.  You may wish to review these *.html files to
get an idea of the various native file formats that Swish-e supports.

You may also use your own test documents.  It's recommended to use small 
test documents when first using Swish-e.

=head2 Step 1: Create a Configuration File

The configuration file controls what and how Swish-e indexes.  The
configuration file consists of directives, comments, and blank lines.
The configuration file can be any name you like.

This example will work with the documents in the F<tests> directory.
You may wish to review the F<tests/test.config> configuration file used
for the C<make test> tests.

For example, a simple configuration file (F<swish-e.conf>):

    # Example Swish-e Configuration file

    # Define *what* to index
    # IndexDir can point to a directories and/or a files
    # Here it's pointing to the current directory
    # Swish-e will also recurse into sub-directories.
    IndexDir .

    # But only index the .html files
    IndexOnly .html

    # Show basic info while indexing
    IndexReport 1

And that's a simple configuration file.  It says to index all the .html
files in the current directory and sub-directories, if any, and provide some
basic output while indexing.

As mentioned above, the complete list of all configuration file directives are described
in L<SWISH-CONFIG|SWISH-CONFIG>.

=head2 Step 2: Index your Files

Run Swish-e using the C<-c> switch to specify the name of the configuration
file.

    swish-e -c swish-e.conf

    Indexing Data Source: "File-System"
    Indexing "."
    Removing very common words...
    no words removed.
    Writing main index...
    Sorting words ...
    Sorting 55 words alphabetically
    Writing header ...
    Writing index entries ...
      Writing word text: Complete
      Writing word hash: Complete
      Writing word data: Complete
    55 unique words indexed.
    4 properties sorted.                                              
    5 files indexed.  1252 total bytes.  140 total words.
    Elapsed time: 00:00:00 CPU time: 00:00:00
    Indexing done!

This created the index file F<index.swish-e>.  This is the default
index file name unless the B<IndexFile> directive is specified in the
configuration file:

    IndexFile ./website.index

You may use the -f switch to specify a index file at indexing time.  The -f 
option overrides a IndexFile setting in the configuration file.

=head2 Step 3: Search

You specify your search terms with the C<-w> switch.  For example, to find
the files that contain the word B<sample> you would issue the command:

    swish-e -w sample

This example assumes that you are in the F<tests> directory. Swish-e returns
in response to that command the following:

    swish-e -w sample

    # SWISH format: 2.4.0
    # Search words: sample
    # Number of hits: 2
    # Search time: 0.000 seconds
    # Run time: 0.005 seconds
    1000 ./test_xml.html "If you are seeing this, the METATAG XML search was successful!" 159
    1000 ./test.html "If you are seeing this, the test was successful!" 437
    .

So the word B<sample> was found in two documents.  The first number
shown is the relevance or rank of the search term, followed by the file
containing the search term, the title of the document, and finally the
length of the document.

The period (".") alone at the end marks the end of results.

Much more information may be retrieved while searching by using
the C<-x> and C<-H> switches (see L<SWISH-RUN|SWISH-RUN>)
and by using Document Properties (see L<SWISH-CONFIG|SWISH-CONFIG>).

=head2 Phrase Searching

To search for a phrase in a document use double-quotes to delimit your
search terms.  (The default phrase delimiter is set in src/swish.h.)

You must protect the quotes from the shell.

For example, under Unix:

    swish-e -w '"this is a phrase" or (this and that)'
    swish-e -w 'meta1=("this is a phrase") or (this and that)'

Or under Windows F<command.com> shell.

    swish-e -w \"this is a phrase\" or (this and that)

The phrase delimiter can be set with the C<-P> switch.

=head2 Boolean Searching

You can use the Boolean operators B<and>, B<or>, or B<not> in searching.
Without these Boolean, Swish-e will assume you're B<and>ing the words together.

Here are some examples:

    swish-e -w 'apples oranges'
    swish-e -w 'apples and oranges'  ( Same thing )

    swish-e -w 'apples or oranges'

    swish-e -w 'apples or oranges not juice' -f myIndex 

retrieves first the files that contain both the words "apples" and "oranges";
then among those the ones that do not contain the word "juice"

A few others to ponder:

    swish-e -w 'apples and oranges or pears'
    swish-e -w '(apples and oranges) or pears'  ( Same thing )
    swish-e -w 'apples and (oranges or pears)'  ( Not the same thing )

Swish processes the query left to right.

See L<SWISH-SEARCH|SWISH-SEARCH> for more information.


=head2 Context Searching

The C<-t> option in the search command line allows you to search for
words that exist only in specific HTML tags.  Each character in the
string you specify in the argument to this option represents a different
tag in which the word is searched; that is you can use any combinations
of the following characters:

    H search in all <HEAD> tags
    B search in the <BODY> tags
    t search in <TITLE> tags
    h is <H1> to <H6> (header) tags
    e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>)
    c is HTML comment tags (<!-- ... -->)

For example:

    # Find only documents with the word "linux" in the <TITLE> tags.
    swish-e -w linux -t t

    # Find the word "apple" in titles or comments
    swish-e -w apple -t tc


=head2 META Tags

As mentioned above, Metanames are a way to define "fields" in your
documents.  You can use the Metanames in your queries to limit the search
to just the words contained in that META name of your document.  For
example, you might have a META tagged field in your documents called
C<subjects> and then you can search your documents for the word "foo" but
only return documents where "foo" is within the C<subjects> META tag.

Document I<Properties> are somewhat related:  Properties
allow the content of a META tag in a source document to be stored within
the index, and that text to be returned along with search results.

META tags can have two formats in your documents.

    <META NAME="keyName" CONTENT="some Content">

And in XML format

    <keyName>
        Some Content
    </keyName>

If using libxml, you can optionally use a non-html tag as a metaname:

    <html>
    <body>
        Hello swish users!
        <keyName>
            this is meta data
        </keyName>.
    </body>

This, of course, is invalid HTML.

To continue with our sample F<Swish-e.conf> file, add the following lines:

    # Define META tags
    MetaNames meta1 meta2 meta3

Reindex to include the changes:

    swish-e -c swish-e.conf

Now search, but this time limit your search to META tag "meta1":

    swish-e -w 'meta1=metatest1'

Again, please see L<SWISH-RUN|SWISH-RUN> and L<SWISH-CONFIG|SWISH-CONFIG>
for complete documentation of the various indexing and searching options.

=head2 Spidering and Searching with a Web form.

This example demonstrates how to spider a web site and setup the included
CGI script to provide a web-based search page.  This example uses Perl
programs included in the Swish-e distribution: F<spider.pl> will be used for
reading files from the web server, and F<swish.cgi> will provide the web
search form and display results.

As an example we will index the Apache Web Server documentation installed on the local
computer at http://localhost/apache_docs/index.html

=over 4

=item 1 Make a Working Directory

Create a directory to store the Swish-e configuration and the Swish-e
index.

    ~$ mkdir web_index
    ~$ cd web_index/
    ~/web_index$

=item 2 Create a Swish-e Configuration file

    ~/web_index$ cat swish.conf 
    # Swish-e config to index the Apache documentation
    #
    # Use spider.pl for indexing (location of spider.pl set at installation time)
    IndexDir spider.pl

    # Use spider.pl's default configuration and specify the URL to spider
    SwishProgParameters default http://localhost/apache_docs/index.html

    # Allow extra searching by title, path
    Metanames swishtitle swishdocpath

    # Set StoreDescription for each parser
    #  to display context with search results
    StoreDescription TXT* 10000
    StoreDescription HTML* <body> 10000

=item 3 Generate the Index

Now run swish-e to create the index:

    ~/web_index$ swish-e -S prog -c swish.conf 

    Indexing Data Source: "External-Program"
    Indexing "spider.pl"
    /usr/local/lib/swish-e/spider.pl: Reading parameters from 'default'

    Summary for: http://localhost/apache_docs/index.html
        Duplicates:     4,188  (349.0/sec)
    Off-site links:       276  (23.0/sec)
           Skipped:         1  (0.1/sec)
       Total Bytes: 2,090,125  (174177.1/sec)
        Total Docs:       147  (12.2/sec)
       Unique URLs:       149  (12.4/sec)
    Removing very common words...
    no words removed.
    Writing main index...
    Sorting words ...
    Sorting 7736 words alphabetically
    Writing header ...
    Writing index entries ...
      Writing word text: Complete
      Writing word hash: Complete
      Writing word data: Complete
    7736 unique words indexed.
    5 properties sorted.                                              
    147 files indexed.  2090125 total bytes.  200783 total words.
    Elapsed time: 00:00:13 CPU time: 00:00:02
    Indexing done!

The above output is actually a mix of output from both swish-e and from spider.pl.
Spider.pl reports the "Summary for: http://localhost/apache_docs/index.html".

Also note that swish-e knows to find spider.pl at /usr/local/lib/swish-e/spider.pl.
The script installation directory (called libexecdir) is set at configure time.
You can see your setting by running swish-e -h:

    ~/web_index$ swish-e -h | grep libexecdir
     Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e

This directory will be needed when setting up the CGI script in the next step.

Finally, verify that the index can be searched from the command line:

    ~/web_index$ swish-e -w installing -m3
    # SWISH format: 2.4.0
    # Search words: installing
    # Removed stopwords: 
    # Number of hits: 17
    # Search time: 0.018 seconds
    # Run time: 0.050 seconds
    1000 http://localhost/apache_docs/install.html "Compiling and Installing Apache" 17960
    718 http://localhost/apache_docs/install-tpf.html "Installing Apache on TPF" 25734
    680 http://localhost/apache_docs/windows.html "Using Apache with Microsoft Windows" 27165
    .

Now try limiting the search to the title:

    ~/web_index$ swish-e -w swishtitle=installing -m3 
    # SWISH format: 2.3.5
    # Search words: swishtitle=installing
    # Removed stopwords: 
    # Number of hits: 2
    # Search time: 0.018 seconds
    # Run time: 0.048 seconds
    1000 http://localhost/apache_docs/install-tpf.html "Installing Apache on TPF" 25734
    1000 http://localhost/apache_docs/install.html "Compiling and Installing Apache" 17960
    .

Note that the above can also be done using the -t option:

    ~/web_index$ swish-e -w installing -m3 -tH

=item 4 Setup the CGI script

Swish-e does not include a web server, therefore you must use your locally
installed web server.  Apache is highly recommended, of course.

Locate your web server's CGI directory.  This may be a cgi-bin directory in your home directory
or a central cgi-bin directory setup by the web server administrator.  Once located copy
the swish.cgi script into the cgi-bin directory.

Where CGI scripts can be located depends completely on the web server used and how it is 
configured.  See your web server's documentation or your site's administrator for additional
information.

This example will use a site cgi-bin directory located at /usr/lib/cgi-bin. 
Copy the swish.cgi script into the cgi-bin directory.  Again, we will need
the location of the libexecdir directory:

    ~/web_index$ swish-e -h | grep libexecdir
     Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e


    ~/web_index$ cd /usr/lib/cgi-bin
    /usr/lib/cgi-bin$ su
    Password: 
    /usr/lib/cgi-bin# cp /usr/local/lib/swish-e/swish.cgi .

If your operating system supports symbolic links, AND your web server allows programs to be 
symbolic links, then you may wish to create a link to the swish.cgi program instead.

    /usr/lib/cgi-bin# ln -s /usr/local/lib/swish-e/swish.cgi

We need to tell the swish.cgi script where to look for the index created in the previous
step.  It's also recommended to enter the path to the swish-e binary, otherwise the
swish.cgi script will look for the binary in the PATH, and that may change when running
under the CGI environment.

Here's the configuration file:

    /usr/lib/cgi-bin# cat .swishcgi.conf 
    return {
        title        => 'Search Apache Documentation',
        swish_binary => '/usr/local/bin/swish-e',
        swish_index  => '/home/moseley/web_index/index.swish-e',
    }


Now, test the script from the command line as a normal user:

    /usr/lib/cgi-bin# exit
    exit

    /usr/lib/cgi-bin$  ./swish.cgi | head
    Content-Type: text/html; charset=ISO-8859-1

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
    <html>
        <head>
           <title>
              Search Apache Documentation
           </title>
        </head>
        <body>

Notice that the CGI script returns the HTTP header (Content-Type) and the body
of the web page, just like a well behaved CGI scrip should do.

Now test using the web server (this step depends on the location of your 
cgi-bin directory).  This example uses the "GET" command that is part of the LWP Perl 
library, but any web browser can run this test.

    /usr/lib/cgi-bin$ GET http://localhost/cgi-bin/swish.cgi | head
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Tranitional//EN">
    <html>
        <head>
           <title>
              Search Apache Documentation
           </title>
        </head>
        <body>
            <h2>

The script reports errors to stderr, so consult the web server's error log if problems
occur.  The message "Service currently unavailable" reported by running swish.cgi
typically indicates a configuration error, and the exact problem will be listed in
the web server's error log.

Detailed instructions on using the F<swish.cgi> script and debugging tips can be found by
running:

    $ perldoc swish.cgi

while in the cgi-bin directory where swish.cgi was copied.  

The spider program F<spider.pl> also has a large number of configuration options.

Documentation is also available in the directory $prefix/share/doc/swish-e or at
http://swish-e.org.

=back

Now you are ready to search.

=head1 Indexing Other Types of Documents

Swish-e can only index HTML, XML and text documents.  In order to index 
other documents such as PDF or MS Word documents you must use a utility to 
convert or "filter" those documents.

There's two ways to filter documents with Swish-e.  Both are described in 
the L<SWISH-CONFIG|SWISH-CONFIG> man page.

The FileFilter directive can be used in your config file to convert 
documents based on their extension.  For example:

    FileFilter .pdf  pdftotext   "'%p' -"
    IndexContents TXT* .pdf

will cause all .pdf files to be filtered through the pdftotext program (part 
of the Xpdf package), and to parse the resulting output from pdftotext with 
the text ("TXT") parser.

The other ways to filter documents is to use a -S prog program and convert 
the documents before passing them onto Swish-e.

For example, spider.pl makes use of a Perl module included with the Swish-e
distribution called SWISH::Filter.  SWISH::Filter is passed a document and
the document's content type and then looks for modules and utilities to
convert the document into one of the types that Swish-e can index.

Swish-e comes ready to index PDF, MS Word, MP3 ID3 tags, and MS Excel file 
types.  But these filters need extra modules or tools to do the actual 
conversion.

For example, the Swish-e distribution includes a module called 
SWISH::Filter::Pdf2HTML that uses the pdftotext and pdfinfo utilities 
provided by the Xpdf package.

This means that if you are using spider.pl to spider your web site and you 
wish to index PDF documents, all that is needed is to install the Xpdf 
package and Swish-e (with the help of spider.pl) will begin indexing your 
PDF files.

=head1 Document Info

$Id: INSTALL.pod,v 1.33 2003/05/21 23:14:03 whmoseley Exp $

.



