&My::Web::footer call is deprecated now, use just: exit;

[www.jankratochvil.net.git] / project / captive / doc / Components.pm

# $Id$
# Captive project doc Components page Perl template.
# Copyright (C) 2003-2005 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; exactly version 2 of June 1991 is required
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA


package project::captive::doc::Components;
require 5.6.0;  # at least 'use warnings;' but we need some 5.6.0+ modules anyway
our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
our $CVS_ID=q$Id$;
use strict;
use warnings;

use My::Web;


sub handler
{
        BEGIN { Wuse 'project::captive::doc::Macros'; }
project::captive::doc::Macros->init(
                "__PACKAGE__"=>__PACKAGE__,
                "title"=>'Captive NTFS Developer Documentation: Components',
                "rel_prev"=>'Architecture.pm',
                "rel_next"=>'Reverse.pm',
                );


print <<"HERE";


<h1>Project Components</h1>

        @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}

        <p>Most of the work of this project is located in the single box called
        &quot;<span class="constant">libcaptive</span>&quot; located in the center
        of the scheme. This component implements the core W32 kernel API by
        various methods described in this document.
        The &quot;<span class="constant">libcaptive</span>&quot; box cannot be
        further dissected as it is just an implementation of a&nbsp;set of
        @{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym','API functions' ]}.
        It could be separated to several subsystems such as the
        @{[ a_href '#cache_manager','Cache Manager' ]},
        Memory Manager, Object Manager, Runtime Library, I/O&nbsp;Manager
        etc. but they have no interesting referencing structure.</p>

        <p>As this project is in fact just a&nbsp;filesystem implementation every
        story must begin at the device file and end at the filesystem operations
        interface. The unified suppported interfaces are
        <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
                (the most low level portability, data-types and utility library for Gnome)
        <span class="type">GIOChannel</span> (for the device access) and the custom
        <span class="constant">libcaptive</span> filesystem API. Each of these ends
        can be connected either to some direct interface (such as the
        <span class="constant">captive-cmdline</span> client),
        @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
        or as a general $GnomeVFS filter.
        @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
        most cases as it offers standard filesystem interface by Linux kernel.
        
        You can also use $GnomeVFS as it offers nice filter interface on
        the UNIX user-privileges level for transparent operation with archives and
        network protocols. This filter interface was used by this project to turn
        the device reference such as <span class="fname">/dev/hda3</span> or <span
        class="fname">/dev/discs/disc0/part3</span> to the fully accessible
        filesystem (pretending being an &quot;archive&quot; in the device
        reference). This device access can be specified by $GnomeVFS URLs such as:
        <span
        class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>

        <p><span class="constant">captive-bug-replay</span> serves just for debugging
        purposes &mdash; you can 'replay' existing
        <span class="fname">file.captivebug.xml.gz</span> automatically being
        generated during W32 filesystem failure. This bugreport file will contain
        all the touched data blocks of the device used in the moment of the
        failure. <span class="constant">captive-bug-replay</span> will therefore
        emulate internal virtual writable device out of these bugreported data.</p>

        <p>If the passed device reference is requested by the user to be accessed
        either in <span class="dashdash">--ro</span> (read-only) mode or in the
        <span class="dashdash">--rw</span> (full read-write) mode there are no
        further device layers needed. Just in the case of <span
        class="dashdash">--blind</span> mode another layer is involved to emulate
        read-write device on top of the real read-only device by the method of
        non-persistent memory buffering of all the possible write requests.</p>

        <p><span class="constant">sandbox commit buffer</span> is involved only in the
        case @{[ a_href 'Details.pm#sandbox','sandboxing feature' ]} is active. It will
        buffer any writes to the device during the sandbox run to prevent
        filesystem damage if the driver would fail in the meantime. If the
        filesystem gets finally successfully unmounted this sandbox buffer can be
        <span id="safe_flush">safely flushed</span>
        to its underlying physical media. The buffer will be dropped
        in the case of filesystem failure, of course. The filesystem should be
        unmounted from time to time &mdash; it can be transparently unmounted and mounted
        by <span class="command">commit</span> of
        <span class="constant">captive-cmdline</span> custom client. Currently you
        cannot force remounting when using
        @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
        but it will be remounted after approx each 1MB data written automatically
        due to @{[ a_href '#log_file_full','NTFS log file full' ]}.

        Now we need to transparently
        @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','connect' ]}
        the device interface of <span class="type">GIOChannel</span> type through
        @{[ a_href 'Details.pm#sandbox','CORBA/ORBit' ]} to the sandboxed slave.</p>

        <p>Such device is still only a&nbsp;UNIX style GLib <span
        class="type">GIOChannel</span> type at this point. As we need to supply it
        to the W32 filesystem driver we must convert it to the W32 I/O&nbsp;Device
        with its capability of handling <span class="type">IRP</span>
                (<span class="constant">I/O Request Packet</span>; structure holding the
                request and result data for any W32 filesystem or W32 block device
                operation)
        requests from its upper W32 filesystem driver. Such W32 I/O&nbsp;Device can
        represent either <span class="type">CD-ROM</span> or
        <span class="type">disk</span> device type as different W32 filesystem
        drivers require different media types &mdash; currently only
        <span class="fname">cdfs.sys</span> requires
        <span class="type">CD-ROM</span> type.</p>

        <p>W32 media I/O&nbsp;Device is accessed from the W32 filesystem driver.
        The filesystem driver itself always creates volume object by
        <span class="function">IoCreateStreamFileObject()</span> representing the
        underlying W32 media I/O&nbsp;Device as the object handled by the
        filesystem driver itself. All the client application filesystem requests
        must be first resolved at the filesystem structures level, passed to the
        volume stream object of the same filesystem and then finally passed to the
        W32 media I/O&nbsp;Device (already implemented by this project as an
        interface to <span class="type">GIOChannel</span> noted above).</p>

        <p>The filesystem driver is called by the core W32 kernel implementation of
        <span class="constant">libcaptive</span> in
        @{[ a_href 'Details.pm#synchronous','synchronous way' ]} in single-shot manner instead of
        the several reentrancies while waiting for the disk I/O completions as can
        be seen in the original
        <span class="productname">Microsoft Windows NT</span>.
        This single-shot synchronous behaviour is possible since all the needed
        resources (disk blocks etc.) can be always presented as instantly ready as
        their acquirement is solved by @{[ a_href 'Architecture.pm#hostosnote','Host-OS' ]} outside of
        the W32 emulated @{[ a_href 'Architecture.pm#guestosnote','Guest-OS' ]} environment.
        For several cases needed only by <span class="fname">ntfs.sys</span> 
        there had to be supported asynchronous access &mdash; parallel execution
        is emulated by GLib <span class="function">g_idle_add_full()</span>
        with <span class="function">g_main_context_iteration()</span> called during
        <span class="function">KeWaitForSingleObject()</span>.</p>

        <p><span class="constant">libcaptive</span> offers the W32 kernel
        filesystem API to the upper layers. This is still not the API the common
        W32 applications are used to as they use W32 libraries which in turn pass
        the call to W32 kernel.  For example
        <span class="function">CreateFileA()</span> is being implemented by several
        libraries such as <span class="fname">user32.dll</span> as a relay
        interface for the kernel function
        <span class="function">IoCreateFile()</span> implemented by this
        project's&nbsp;<span class="constant">libcaptive</span> W32 kernel
        emulation component.</p>

        <p>As it would be very inconvenient to use the legacy, bloated and UNIX
        style unfriendly W32 kernel filesystem API this project offers its own
        @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
        the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
        This interface is supposed to be easily utilized by
        <a href="#client_interface_customapp">a&nbsp;custom application accessing
        the W32 filesystem driver</a>.</p>

        <p>@{[ a_href 'Details.pm#sandbox','CORBA/ORBit' ]} hits us again &ndash; we need to
        @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','translate' ]}
        the @{[ a_href '#client_interface','custom filesystem API interface' ]}
        out of the sandboxed slave to the UNIX space.</p>

        <p><span class="constant">captive sandbox master</span> provides the
        functionality of covering any possible sandboxed slave restarts and its
        communication. It is also capable of
        <span id="demultiplexing_master">demultiplexing single API operations</span>
        to multiple its connected sandbox slaves in transparent way
        as each of them handles
        @{[ a_href 'Details.pm#mounted_one','just one filesystem device' ]}.</p>

        <p>The rest of the story is not much special for this project since this is
        a common UNIX problem how to offer user space implemented UNIX filesystem
        as a generic system filesystem (as those are usually implemented only as
        the components od UNIX kernel).</p>

        <p>The filesystem service can be offered in several ways:</p>

        <dl>
                <dt>Custom client</dt>
                <dd>
                        <p>One possibility would be to write
                        <span id="client_interface_customapp">a custom client application</span>
                        for this project such as file manager or a&nbsp;shell. Although it
                        would implement the most appropriate user interface to the set of
                        functions offered by this project (and W32 filesystem API) it has the
                        disadvantage of special client software. Appropriate client is provided
                        by this project as:
                        <span class="fname">src/client/cmdline/cmdline-captive</span></p>
                </dd>

                <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
                <dd>
                        <p>The most usable interface is the
                        @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
                        by <span class="constant">liblufs-captivefs</span>.
                        As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
                        already assigns separate process for each filesystem mount the
                        @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
                        is not utilized in this case.</p>

                        <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
                        needs multiple operating threads (each UNIX kernel operation needs
                        one free lufsd slot/thread to not to fail immediately).
                        As <span class="constant">libcaptive</span> is
                        @{[ a_href 'Details.pm#synchronous','single-threaded' ]} all the operations
                        get always synchronized by
                        <span class="constant">liblufs-captivefs</span>
                        before their pass over to <span class="constant">libcaptive</span>.</p>
                </dd>

                <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
                <dd>
                        <p>This client allowing its filesystem access even without any
                        involvement of UNIX kernel from any $GnomeVFS aware client application
                        (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
                        This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
                        data flow of this project in two points &mdash; both as the lowest layer
                        device image source and also as the upper layer for the filesystem
                        operation requests.</p>
                </dd>
        </dl>

        <p>Unimplemented and deprecated methods for providing filesystem
        service:</p>

        <dl>
                <dt>W32 filesystem in UNIX OS kernel</dt>
                <dd>
                        <p>The real UNIX OS filesystem implementation must be completely
                        implemented inside the hosting OS kernel. This requires special coding
                        methods with limited availability of coding features and libraries.
                        Also it would give the full system control to the untrusted W32
                        filesystem driver code with possibly fatal consequences of yet
                        unhandled W32 emulation code paths. It would benefit from the best
                        execution performance but this solution was never considered a real
                        possibility.</p>
                </dd>

                <dt>Custom NFS server</dt>
                <dd>
                        <p>The common approach
                        <span id="offered_NFS">of filesystem implementations</span>
                        outside UNIX OS kernel were custom NFS servers usually running on the
                        same machine as the NFS-connected client as such NFS server is usually
                        an ordinary UNIX user space process. It would be possible to implement
                        this project as a&nbsp;custom NFS server but the NFS protocol itself
                        has a&nbsp;lot of fundamental flaws and complicated code for backward
                        compatibility.</p>
                </dd>
        </dl>


HERE


exit;
}
1;