Devil Killer Is Here MiNi Shell
MiNi SheLL

Current Path : /proc/thread-self/root/usr/local/man/man3/
Linux boscustweb5006.eigbox.net 5.4.91 #1 SMP Wed Jan 20 18:10:28 EST 2021 x86_64
Current File : //proc/thread-self/root/usr/local/man/man3/KinoSearch::Docs::DevGuide.3
.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sh \" Subsection heading
.br
.if t .Sp
.ne 5
.PP
\fB\\$1\fR
.PP
..
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  | will give a
.\" real vertical bar.  \*(C+ will give a nicer C++.  Capital omega is used to
.\" do unbreakable dashes and therefore won't be available.  \*(C` and \*(C'
.\" expand to `' in nroff, nothing in troff, for use with C<>.
.tr \(*W-|\(bv\*(Tr
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
'br\}
.\"
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.if \nF \{\
.    de IX
.    tm Index:\\$1\t\\n%\t"\\$2"
..
.    nr % 0
.    rr F
.\}
.\"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.hy 0
.if n .na
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "KinoSearch::Docs::DevGuide 3"
.TH KinoSearch::Docs::DevGuide 3 "2006-12-27" "perl v5.8.8" "User Contributed Perl Documentation"
.SH "NAME"
KinoSearch::Docs::DevGuide \- hacking/debugging KinoSearch
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
Developer-only documentation.  If you just want to build a search engine, you
probably don't need to read this.
.SH "Fundamental Classes"
.IX Header "Fundamental Classes"
Most of the classes in KinoSearch rely on 
KinoSearch::Util::Class and
KinoSearch::Util::ToolSet, so you'll probably
want to familiarize yourself with them.
.SH "Object Oriented Design"
.IX Header "Object Oriented Design"
.Sh "No public member variables."
.IX Subsection "No public member variables."
Multiple classes defined within a single source-code file, e.g. TermQuery and
TermWeight, may use direct access to get at each others member variables.
Everybody else has to use accessor methods.
.PP
C\-struct based classes such as TermInfo allow direct access to their members,
but only from C (of course).
.Sh "Subroutine/method access levels"
.IX Subsection "Subroutine/method access levels"
There are three access levels in KinoSearch.  
.IP "1" 4
.IX Item "1"
\&\fBpublic\fR: documented in \*(L"visible\*(R" pod.
.IP "2" 4
.IX Item "2"
\&\fBprivate\fR: subs which are prepended with an _underscore may only be used
within the package in which they reside \*(-- as per perlstyle
guidelines \*(-- and in only one source file.
.IP "3" 4
.IX Item "3"
\&\fBdistro\fR: any sub which doesn't fall into either category above may be used
anywhere within the KinoSearch distribution.  
.SH "Documentation Conventions"
.IX Header "Documentation Conventions"
KinoSearch's public \s-1API\s0 is defined by what you get when you run the suite
through a well-behaved pod-to-whatever converter.  Developer-only
documentation is limited to comments and \*(L"invisible\*(R" =for/=begin \s-1POD\s0 blocks.
.SH "Integration of XS and C code"
.IX Header "Integration of XS and C code"
\&\s-1XS\s0 and C code in KinoSearch is stored faux\-Inline\-style, after an
\&\f(CW\*(C`_\|_END_\|_\*(C'\fR token, and delimited by either \f(CW\*(C`_\|_XS_\|_\*(C'\fR, \f(CW\*(C`_\|_H_\|_\*(C'\fR, or \f(CW\*(C`_\|_C_\|_\*(C'\fR.  A
heavily customized Build.PL detects these code blocks and writes out hard
files at install\-time, so the inlining is mostly for convenience while
editing: the \s-1XS\s0 code is often tightly coupled to the Perl code in a given
module, and having everything in one place makes it easier to see what's going
on and move things back and forth.
.PP
Build.PL writes out separate .h and .c files for each block it finds, but all
the \s-1XS\s0 blocks are concatenated into a single file \*(-- KinoSearch.xs.  The
content of KinoSearch.xs consists of the \s-1XS\s0 block from KinoSearch.pm, followed
by all the other \s-1XS\s0 blocks in an undetermined order.  Ultimately, only a
single compiled library gets installed along with the Perl modules.
.PP
At runtime, the only module which calls XSLoader::load is KinoSearch.  Because
the KinoSearch \f(CW\*(C`MODULE\*(C'\fR has many \f(CW\*(C`PACKAGE\*(C'\fRs, \f(CW\*(C`use KinoSearch;\*(C'\fR loads \fIall\fR
of the \s-1XS\s0 routines in the entire KinoSearch suite.  A pure-Perl version of
KinoSearch.pm which did the same thing might look like this...
.PP
.Vb 2
\&    package KinoSearch;
\&    our $VERSION = 1.0;
.Ve
.PP
.Vb 1
\&    package KinoSearch::Index::TermInfo;
.Ve
.PP
.Vb 3
\&    sub get_doc_freq {
\&        # ...
\&    }
.Ve
.PP
.Vb 1
\&    package KinoSearch::Store::InStream;
.Ve
.PP
.Vb 3
\&    sub lu_read {
\&        # ...
\&    }
.Ve
.PP
.Vb 1
\&    # ...
.Ve
.PP
Since KinoSearch.xs is only generated/modified when Build.PL is run, an extra
command line call to Build.PL has to be integrated into the development
workflow when working on \s-1XS\s0 or C material.
.PP
.Vb 1
\&    % perl Build.PL; ./Build code; perl -Mblib t/some_test.t
.Ve
.PP
Build.PL tracks modification times, using them to determine whether it needs
to recompile anything. If only pure Perl modules have been edited, it won't
force needless recompilation, and if only a limited number of .pm files
containing \s-1XS/C/H\s0 code have been edited, it will recompile as little as it
can.
.Sh "Divison of labor between Perl, \s-1XS\s0, and C"
.IX Subsection "Divison of labor between Perl, XS, and C"
Technically, most or all of the C code in KinoSearch could be stuffed into \s-1XS\s0
functions.  However, the mechanics of moving data across the Perl/C boundary
are complicated, and the added cruft tends to make what's going on in the C
code more difficult to grok.
.PP
To maximize clarity, when possible \s-1XS\s0 in KinoSearch is limited to \*(L"glue\*(R"
code, while Perl and C do the heavy lifting.  Exceptions occur when \s-1XS\s0
functions need to manipulate the Perl stack, for instance when returning more
than one value.
.SH "Relationship to Lucene"
.IX Header "Relationship to Lucene"
.Sh "\s-1API\s0 Differences, both public and private"
.IX Subsection "API Differences, both public and private"
Since there is no method overloading by signature in Perl, it's impossible for
KinoSearch to mimic Lucene's \s-1API\s0 exactly.  For a variety of other reasons,
most of them performance\-related, it's difficult or inadvisable to even try.
The most crucial \s-1API\s0 distinction between KinoSearch and Lucene relates to
indexing: KinoSearch, like its predecessor, Search::Kinosearch, forces you to
specify field definitions in advance; in contrast, Lucene employs an elaborate
apparatus behind the scenes which adapts field definitions on the fly.  If
KinoSearch were to implement that apparatus, indexing would be substantially
slower than it is.  
.PP
Since KinoSearch could only muster a poor imitation of Lucene's \s-1API\s0 at best,
it doesn't work very hard at imitating it \*(-- it just tries to be a good,
idiomatic Perl search \s-1API\s0.  Here's a sampling of other ways in which
KinoSearch and Lucene differ:
.IP "\(bu" 4
In Lucene both IndexWriter and IndexReader can be used to modify an index,
while in KinoSearch all index modifications are performed via the InvIndexer
class.  
.IP "\(bu" 4
Since Perl's filehandles can be used to write to scalars, there's little
benefit to having separate classes like RAMIndexOutput and FSIndexOutput, and
KinoSearch only has two \s-1IO\s0 classes: InStream and OutStream.
.IP "\(bu" 4
Certain classes have had their names transformed in KinoSearch, usually by
shortening (SegmentTermDocs => SegTermDocs, Document => Doc), but sometimes
arbitrarily (Directory => InvIndex).  
.IP "\(bu" 4
Other classes are specific to KinoSearch, e.g. PostingsWriter, PolyAnalyzer.
These are either scavenged from Search::Kinosearch, or written from scratch.
.IP "\(bu" 4
A lot of the common method names in Lucene are core keywords in Perl: length,
close, seek, read, write.  Where it seemed important to disabiguate those,
they have been changed \*(-- e.g., for a Perl programmer accustomed to the
passive behavior of CORE::close, having a \fIclose()\fR method that triggers
file-writes is counter\-intutive, so such behaviors have typically been shunted
into a method called \fIfinish()\fR.
.PP
Nevertheless, developers who speak both Perl and Java shouldn't find it too
difficult to move back and forth between Lucene and KinoSearch \*(-- as long as
what they want to do is within the scope of KinoSearch's leaner \s-1API\s0.  At the
time of this writing KinoSearch's public \s-1API\s0 is artificially small because
some pieces aren't done, and it's a young library so preserving the
flexibility to change the internal implementation is paramount.  But
KinoSearch does not aspire to do all the things that  Lucene does \*(-- only the
things that can be done well, and maybe a few tricks of its own.
.Sh "Thematic architectural differences"
.IX Subsection "Thematic architectural differences"
The relatively low object-oriented overhead of Java makes possible some
architectures in Lucene which do not translate well to Perl.  For instance,
arrays are often accumulated in Lucene by calling an iterator's \fInext()\fR
method over and over.  Method calls in Perl, which are always dynamic, are
more costly, making this technique impractical in areas which are both
data-intensive and performance\-critical.  Most often, KinoSearch solves the
problem by moving the algorithm to a for/while loop in either Perl or C.  In a
minority of cases, frequently called methods are emulated using either a
globally exposed C function or C pointer\-to\-function.
.PP
Lucene is also a profligate wastrel when it comes to creating and destroying
objects \*(-- it can afford to be, because Java objects come cheap.  For example,
during the indexing phase Lucene creates a mini-inverted-index for each
document, complete with its own FieldsWriter, DocumentWriter, and so on.
Having Perl objects do the same thing in the same way yields disappointing
execution speed.  Therefore, KinoSearch does the same thing, but in a
different way: it uses a completely different merge model, based around
serialization and external sorting, which makes it possible to
write indexes by-the-segment rather than by-the document \*(-- slashing 
the \s-1OO\s0 requirements and largely offsetting the extra overhead.
.PP
Similar architectural differences exist throughout KinoSearch.  While the
file-format dictates certain algorithms, when idiomatic alternatives presented
themselves or were already known, they were adopted.
.Sh "Search Logic"
.IX Subsection "Search Logic"
KinoSearch's search logic differs from Lucene's.  In particular,
very short fields do not contribute as much to a score, making it essential to
assign a <boost> of greater than 1 to short but important fields, e.g.
\&\*(L"title\*(R".  
.Sh "File Format"
.IX Subsection "File Format"
Lucene's file format is a beautiful thing.  KinoSearch was retooled to use it
because it didn't seem possible to improve on it more than superficially.
It's somewhat hard to work with, and impossible to work with efficiently using
only native Perl, but it has both data-compression and
search-time-execution-speed nailed.
.PP
Unfortunately, the way Lucene defines a String has a Java-centric quirk that
deep-sixes prospects for 100% index compatibility, and the only realistic
solutions involve changes to Lucene.  In theory, these changes (moving to
bytecount-based Strings) should actually improve Lucene's performance, and the
Lucene development community has expressed openness to them.  At some point,
time may be found to write these win-win patches, but at present working on
KinoSearch itself takes higher priority.
.PP
Here's a list of the ways in which KinoSearch's file format differ's from
Lucene's:
.IP "\(bu" 4
\&\fBStrings\fR \- Lucene's file format specifies that a String is a VInt count of
the number of java chars a string occupies, followed by the character data
encoded in Sun's \*(L"modified \s-1UTF\-8\s0\*(R" format (a misnamed, illegal variant,
unsanctioned by the Unicode Consortium).  Dealing with this format in Perl has
proven to be so fiddly and performance-killing as to be not worth the effort.
At present, KinoSearch's String format is: a byte count, followed by arbitrary
data.  
.IP "\(bu" 4
\&\fBField Numbers\fR \- KinoSearch requires that field numbers correspond to
lexically sorted order of field names; Lucene indexes are unlikely to have
this property.  KinoSearch 0.05 had compatibility code which dealt with
unordered field numbers, but that code was removed in 0.06.  It can be
restored if need be.
.IP "\(bu" 4
\&\fBTerm Vectors\fR \- KinoSearch groups term vector data with stored field data;
Lucene uses separate files.  Keeping term vector data and field data together
minimizes disk activity for common usage patterns.  It's also a little simpler
to maintain.  However, if the String issue is resolved in a future version of
Lucene, KinoSearch will likely switch to Lucene's format.
.SH "Coding style"
.IX Header "Coding style"
The gist: when possible, KinoSearch's Perl code follows the recommendations
set out in Damian Conway's book, \*(L"Perl Best Practices\*(R", and its \s-1XS/C\s0 code more
or less (no tab characters allowed) emulates the style of the source
code for Perl itself.
.PP
Perl code is auto-formatted using a PerlTidy-based helper app called kinotidy,
which is basically perltidy with a profile set up to use the \s-1PBP\s0 settings.  
.PP
It would be nice if there were a formatter for \s-1XS\s0 and C code that was as good
as PerlTidy.  Since there isn't, the code is manually set to look as though it
was, with one important difference: a bias towards maximum parenthetical
tightness.  
.PP
In both Perl and \s-1XS/C\s0, code is organized into commented paragraphs a few lines
in length, as per \s-1PBP\s0 recommendations.  Strong efforts are made to keep the
comment to a single line.  Stupefyingly obvious \*(L"code narration\*(R" comments are
used when something more literate doesn't present itself \*(-- the goal is to
be able to grok the intended flow of a function by scanning the first line of
each \*(L"paragraph\*(R".
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 2005\-2006 Marvin Humphrey
.SH "LICENSE, DISCLAIMER, BUGS, etc."
.IX Header "LICENSE, DISCLAIMER, BUGS, etc."
See KinoSearch version 0.15.
Creat By MiNi SheLL
Email: devilkiller@gmail.com