&My::Web::footer call is deprecated now, use just: exit;

[www.jankratochvil.net.git] / project / captive / doc / Details.pm

# $Id$
# Captive project doc Details page Perl template.
# Copyright (C) 2003-2005 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; exactly version 2 of June 1991 is required
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA


package project::captive::doc::Details;
require 5.6.0;  # at least 'use warnings;' but we need some 5.6.0+ modules anyway
our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
our $CVS_ID=q$Id$;
use strict;
use warnings;

use My::Web;


sub handler
{
        BEGIN { Wuse 'project::captive::doc::Macros'; }
project::captive::doc::Macros->init(
                "__PACKAGE__"=>__PACKAGE__,
                "title"=>'Captive NTFS Developer Documentation: Implementation Details',
                "rel_prev"=>'CacheManager.pm',
                "rel_next"=>'APITypes.pm',
                );


print <<"HERE";


<h1>Implementation Details</h1>

        <h2 id="emulmeth">Choice of the Emulation Methods</h2>

                <p>The intent of the project was to get reliable read-write access to
                <span class="productname">NTFS</span> partition. There are several possible
                ways to achieve that:</p>

                <h3 id="emulmeth_vm">Virtualmachine Running the Original W32 Subsystem</h3>

                        <p>Creating virtual-hardware PC and running the original W32 binaries
                        including their boot-loader etc. Disk device access would be passed as
                        virtual IDE disk (=hard disk drive). File access API would be implemented
                        either by special escaping by some trapped instruction out of the
                        virtualmachine while using W32 file access API or using the standard W32
                        SMB (Server Message Block) network access through some virtual network
                        card. The latter network access solution is almost the currently available
                        possibility of running full-blown disk-sharing real
                        <span class="productname">Microsoft Windows NT</span> inside virtual
                        machine emulator such as <span class="productname">VMware</span>.</p>

                        <p>pros: Full compatibility due to fully native codebase.</p>

                        <p>cons: Hard to debug, missing documentation of NT booting internals,
                        possible problems by different PC virtual-hardware than expected by NT,
                        requirement of fully installed
                        <span class="productname">Microsoft Windows NT</span> product.</p>

                <h3 id="method_ntoskrnl">&quot;ntoskrnl.exe&quot; Inside Virtual Address Space</h3>

                        <p>This solution was chosen by the project. Binary filesystem driver and
                        also <span class="fname">ntoskrnl.exe</span> binary file are required.
                        Unfortunately <span class="fname">ntoskrnl.exe</span> expects a&nbsp;native
                        PC virtual-hardware missing during regular UNIX user space process
                        emulation, therefore such instructions must be trapped and emulated/ignored
                        from case to case.</p>

                        <p>Also the <span id="init_ntoskrnl">initialization code of <span
                        class="fname">ntoskrnl.exe</span></span> is not executed by this project since
                        it expects to get full PC hardware access privileges and thus some
                        datastructures do not get initialized by it (need to be trapped later at
                        runtime stage). Some of the missing initializations are solved by
                        @{[ a_href 'APITypes.pm#functype_wrap','API functions wrapping' ]}.</p>

                        <p>pros: Lightweight, easier to debug.</p>

                        <p>cons: Possible incompatible emulation of
                        <span class="fname">ntoskrnl.exe</span> parts, missing documentation needed
                        for the implementation.</p>

                <h3 id="emulmeth_fs">Filesystem Driver Inside Virtual Address Space</h3>

                        <p>Unlike @{[ a_href 'Details.pm#method_ntoskrnl','previous method' ]} here we do not use
                        even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
                        W32 is <span id="native_ntoskrnl">emulated from the project source
                        files</span>. <span class="fname">cdfs.sys</span> driver was successfuly ran
                        in this manner in the former versions of this project but the possibility
                        to run without <span class="fname">ntoskrnl.exe</span> was dropped since it
                        had no licensing gains (you need the original
                        <span class="productname">Microsoft Windows NT</span> files at least for
                        the filesystem driver itself) and the emulation of undocumented parts
                        reusable from <span class="fname">ntoskrnl.exe</span> binary was
                        a&nbsp;pain.</p>

                        <p>pros: Lightweight, easier to debug.</p>

                        <p>cons: Possible incompatible emulation of the whole
                        <span class="fname">ntoskrnl.exe</span>, its missing documentation.</p>


        <h2 id="apichoice">API Function Implementation Choices</h2>

                <p>During the initial point of the project development all the API
                functions were defined as unimplemented, of course. Any call of such
                unimplemented function is fatal and results in program termination. When we
                need to implement any required API function we have multiple choices to do
                so:
                @{[ a_href 'APITypes.pm#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
                @{[ a_href 'APITypes.pm#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
                @{[ a_href 'APITypes.pm#functype_native_reactos','Native implementation &ndash; $ReactOS' ]},
                @{[ a_href 'APITypes.pm#functype_native_wine','Native implementation &ndash; $Wine' ]}
                or
                @{[ a_href 'APITypes.pm#functype_native_libcaptive','Native implementation &ndash; project specific' ]}.
                <!-- a_href 'APITypes.pm#functype_undef','Undefined function' -->
                </p>


        <h2 id="sandbox">Sandboxing of W32 Filesystem</h2>

                <p>The emulated W32 environment running the original W32 filesystem driver
                is separated from the rest of UNIX OS. It achieves the following goals:</p>

                <ul>
                        <li><b>Restartable</b>: W32 driver can be restartde in clean state if it crashed</li>
                        <li><b>Secure</b>: Malicious W32 code cannot affect the security of UNIX OS</li>
                        <li><b>Stable</b>: Buggy W32 cannot crash any part of UNIX OS</li>
                </ul>

                <p>Sandboxing is provided with the following attributes:</p>
                
                <ul>
                        <li>standalone UNIX process with separate memory space</li>
                        <li>chroot(2) in empty directory to prevent any UNIX OS filesystem access</li>
                        <li>setuid(2) to own user/group to prevent interaction with UNIX processes</li>
                        <li>setrlimit(2) to limit system resources available for W32 environment</li>
                        <li>the only connection with the UNIX OS by CORBA/ORBit RPC</li>
                </ul>

                @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}

                <p>This security is almost the same as provided by
                emulated virtual machines such as
                @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware' ]}.</p>

                @{[ doc_img 'dia/inheritance','Sandboxing Scheme' ]}

                <p>Project can be also used in non-sandboxed mode by
                <span class="command">--no-sandbox</span> option as it is easier to debug
                without CORBA/ORBit RPC. In this case the
                <span class="type">DirectorySlave</span>/<span class="type">FileSlave</span>
                options are used directly instead of their
                <span class="type">DirectoryParent</span>/<span class="type">FileParent</span>
                peers.</p>


        <h2 id="patched">&quot;patched&quot; vs. &quot;unpatched&quot; Libraries</h2>

                <p>Library is called <span class="constant">patched</span> if we require
                loading its original binary code file. Project needs to patch it to be able
                to trap all the function entry points. The only currently
                <span class="constant">patched</span> library of this project is
                <span class="fname">ntoskrnl.exe</span>.</p>

                <p>Library is called <span class="constant">unpatched</span> if no original
                binary code is needed since all of its functions are completely emulated by
                @{[ a_href 'APITypes.pm#functype_native','the native implementations' ]} of this project.
                The typical <span class="constant">unpatched</span> representative is
                <span class="fname">hal.dll</span> as it specializes on the hardware
                dependent code and therefore it must be completely replaced by this project
                running in the $gnulinux operating system environment. Early versions of
                this project had also full <span class="constant">unpatched</span>
                <a href="#native_ntoskrnl">native implementation of
                <span class="fname">ntoskrnl.exe</span></a> but it no longer applies.</p>

        <h2 id="mman">Memory Management</h2>

                <p>Original <span class="productname">Microsoft Windows NT</span>
                architecture uses two address space areas &ndash; user space and kernel space.
                User space is mapped in the range <span class="constant">0x00000000</span>
                to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
                range <span class="constant">0x80000000</span>
                (<span class="constant">KERNEL_BASE</span> in $ReactOS sources) to
                <span class="constant">0xFFFFFFFF</span>. All these virtual memory ranges
                represent addresses after their MMU (Memory Management Unit) mapping, of
                course. More discussion can be found in the
                <a href="http://www.microsoft.com/hwdev/platform/server/PAE/PAEmem.asp">description 
                by <span class="productname">Microsoft</span></a>.</p>

                <p>This project runs in the virtual address space used both for the UNIX
                user space process part and for the W32 kernel space. Therefore this
                project defines that W32 kernel runs in the whole range
                <span class="constant">0x00000000</span> to
                <span class="constant">0xFFFFFFFF</span> since there are no special mapping
                assumptions about the UNIX user space process mapping. No W32 user space
                exists in this project. Such approach also nullifies any special memory
                moving operations between W32 kernel space and W32 user space memory areas
                (such as <span class="function">MmSafeCopyToUser()</span>).</p>

        <h2 id="unicode">Unicode Strings and Characters</h2>

                <p>W32 platform uses 16-bit type <span class="type">wchar_t</span> while $gnulinux uses a
                32-bit one. This can be problem during GCC (GNU C&nbsp;Compiler)
                compilation of combination of native UNIX C&nbsp;sources (assuming 32-bit
                GCC with 32-bit <span class="type">wchar_t</span>) and
                $ReactOS C sources (assuming W32 compiler with 16-bit
                <span class="type">wchar_t</span>) for literal wide strings
                (C source file systax: <span class="command">L&quot;wstring&quot;</span>).
                Possibilities to solve this issue list:</p>

                <ul>
                        <li>
                                <p>Using <span class="constant">-fshort-wchar</span> GCC option and
                                strictly differentiate between compilation of
                                <span class="productname">ReactOS</span> code and UNIX code.</p>

                                <p>pros: No source modifications needed, no runtime performance hit.</p>

                                <p>cons: No type checking if some part of code has bad compilation
                                flags, complicated way to completely split
                                <span class="productname">ReactOS</span> and UNIX code.</p>
                        </li>
                        <li>
                                <p>Wrap all <span class="productname">ReactOS</span> literal constants
                                by some conversions function call (implemented as macro
                                <span class="function">REACTOS_UCS2()</span> by this project).</p>

                                <p>pros: Any forgotten/mistaken conversions are type-checked and warned
                                during the compilation by GCC.</p>

                                <p>cons: All compiled <span class="productname">ReactOS</span> sources
                                files containing literal wide strings have to be wrapped/modified,
                                performance hit by runtime string conversions.</p>

                                <p>This solution was chosen to get the internal sanity checking
                                benefit.</p>
                        </li>
                </ul>

        <h2 id="binfmt">Supported Binary Formats</h2>

                <p>The native W32 binary format is identified as
                <span class="constant">PE-32</span> (Portable Executable 32-bit), such
                files have all the usual extensions such as
                <span class="fname">.sys</span>, <span class="fname">.exe</span>,
                <span class="fname">.dll</span> etc. <span class="constant">PE-32</span>
                loading support was already implemented by $ReactOS, its memory mapping
                specifics just had to be ported to $gnulinux environment by this project.
                This loading support does not (yet) cover importing of debug symbols from
                W32 <span class="fname">.PDB</span> (Program DataBase) files in $gnulinux
                ABI (Application Binary Interface) compatible way.</p>

                <p>This project also supports transparent loading of UNIX
                <span class="fname">.so</span> (Shared Object file) binary format. If you
                have W32 source files for some W32 library you can try to compile it by GCC
                to get the shared library with $gnulinux ABI compatible debug information
                (GCC option <span class="constant">-ggdb3</span> recommended). Beware of
                possible compilation problems as <span class="productname">Microsoft</span>
                C&nbsp;code expects <span class="constant">exception</span> handling to be
                supported by the compiler (definitely not the case of the plain C compiler
                of GCC) &mdash; all the exception catching code should be discarded as any
                @{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
                such driver is running in the scope of this project. You can use the
                following script of this project to compile W32 filesystem source files as
                UNIX <span class="fname">.so</span>:
                @{[ captive_srcfile 'src/w32-mod/ext2fsd.so-build.sh' ]}</p>
                
                <p>Be aware of some differences if you use
                <span class="constant">PE-32</span> binary format file vs.
                <span class="fname">.so</span> format file.
                <span class="constant">PE-32</span> use the appropriate W32 specific
                @{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
                <span class="fname">.so</span> must be completely compiled in the standard
                UNIX @{[ a_href 'CallType.pm#calltype_cdecl','cdecl call type semantics' ]}.
                @{[ a_href 'APITypes.pm#functype_native','Native function implementations' ]} do not need
                to be explicitely exported by <span class="fname">captivesym</span> as they
                are resolved automatically by the UNIX dynamic system linker. It may be
                surprising you will have to fix all such missing symbol exports if you
                advance during the development from the debugging
                <span class="fname">.so</span> file for the production version of the
                original <span class="constant">PE-32</span> binary file.</p>


        <h2 id="mounted_one">At Most One Mounted Filesystem</h2>

                <p>The project technically supports only one (exactly one...) mounted
                filesystem device and only one filesystem driver. There is nothing
                complicated to support multiple disks and multiple loaded filesystem
                modules but as they would share the address space it would only bring
                a&nbsp;possible complications during bug reports and the bug solving
                itself.  It was considered as a&nbsp;more sane way to support multiple W32
                mounted disks by completely separately running project instances in
                a&nbsp;different UNIX processes communicating from their sandboxes via
                @{[ a_href 'Details.pm#sandbox','CORBA sandbox interface' ]}. This sandboxing
                feature is not yet deployed although its code is already prepared.</p>

                <p>The project also does not support any state cleanup to be able to load
                filesystem&nbsp;<span class="constant">A</span>,
                cleanup&nbsp;<span class="constant">A</span> and load a different
                filesystem&nbsp;<span class="constant">B</span> in the same process address
                space. It complies with the preventions of the possible debugging
                complications as noted above. Despite this you still must call the function
                <span class="function">captive_shutdown()</span> to flush all the pending
                filesystem buffers to the disk. After calling
                <span class="function">captive_shutdown()</span> the process address space is
                no longer usable for any further project operations and the process is
                expected to be terminated in the manner compatible with its driving
                @{[ a_href 'Details.pm#sandbox','CORBA sandbox interface' ]} control master.</p>

                <p>Each sandbox executing the untrusted W32 binary filesystem driver code
                is connected through its
                @{[ a_href 'Details.pm#sandbox','CORBA sandbox interface' ]} at the point of upper
                layer <span class="constant">libcaptive</span>-specific filesystem API, at
                the point of the bottom layer of <span class="type">GIOChannel</span>
                device access and also for transfers of GLib logging
                messages/warnings/errors out of the sandbox to the user.</p>


        <h2 id="synchronous">Multithreading and Multiple Processors</h2>

                <p>W32 platform stands on its&nbsp;thorough architecture parallelism. It
                must lock all its objects to maintain coherence in presence of
                multithreading and multiple processors. Since the author of this project
                considers any parallel execution a serious obstacle for debugging the whole
                project architecture was designed to prevent any undeterministic behaviour.
                Therefore this projects always emulates uniprocessor
                <span class="productname">Microsoft Windows NT</span> kernel
                (<span class="constant">KeNumberProcessors</span> symbol is always 1),
                everything runs in the single initial thread/process and all the filesystem
                operations are performed as synchronous
                        (&quot;synchronous&quot; by flags
                        <span class="constant">FILE_SYNCHRONOUS_IO_ALERT</span>,
                        <span class="constant">FO_SYNCHRONOUS_IO</span>,
                        <span class="constant">IRP_SYNCHRONOUS_API</span>,
                        <span class="constant">IRP_SYNCHRONOUS_PAGING_IO</span>,
                        forced <span class="constant">TRUE</span> result of
                        <span class="function">IoIsOperationSynchronous()</span>
                        etc.).
                For several cases needed only by <span class="fname">ntfs.sys</span> there
                had to be supported asynchronous access
                (<span class="constant">STATUS_PENDING</span> return code) &ndash; parallel
                execution is emulated by GLib
                <span class="function">g_idle_add_full()</span> with
                <span class="function">g_main_context_iteration()</span> called during
                <span class="function">KeWaitForSingleObject()</span>.</p>
                <p>Since there is a&nbsp;possibility a&nbsp;real W32 parallel threading would
                be yet needed in the future all the code that would be hit by W32
                multithreading capability is marked by
                <span class="constant">TODO:thread</span> comment.</p>

                <p>Multiple processors (SMP) support will never need to be implemented
                since uniprocessor W32 kernels apparently run the filesystem driver modules
                fine. As this project implements only the uniprocessor W32 kernel all the
                processor locking functions and structures such as
                <span class="constant">KSPIN_LOCK</span> etc. can be safely implemented as
                no-operations.</p>

                <p>Asynchronous callbacks registered for
                <span class="constant">IO_WORKITEM</span>s are passed as GLib idle
                functions by <span class="function">g_idle_add_full()</span>. Although they
                will probably never be executed during non-interactive project's batch
                executions it is the&nbsp;responsibility of W32 driver implementation to
                complete all the pending tasks before its W32 shutdown. Such W32 shutdown
                is done during cleanup of the project's&nbsp;execution by
                <span class="function">captive_shutdown()</span>.</p>

        <h2 id="paranoia">Paranoia Checks</h2>

                <p>A&nbsp;general approach of software projects development is to implement
                many internal sanity checks during the development stage but to produce the
                most optimized final release product without those debugging checks.</p>

                <p>Facilities for these practices can be seen in the standard
                C&nbsp;include files for example as function
                <span class="function">assert()</span> which gets disabled by the
                <span class="constant">NDEBUG</span> symbol used during the final optimized
                executable compilation. This project uses Gnome GLib messaging subsystem
                offering sanity checks discarded by symbols
                <span class="constant">G_DISABLE_ASSERT</span> and
                <span class="constant">G_DISABLE_CHECKS</span>.
                <span class="productname">Microsoft</span> also produces two versions of
                its products &ndash; regular customers use the &quot;free build&quot; (also
                called &quot;retail&quot;) while the programmers should develop their code
                on the &quot;checked build&quot; product releases.</p>

                <p>As this project will always run unknown binary code of proprietary W32
                filesystem drivers, the code can never be trusted. Such code even runs in
                the same unprotected address space as its controlling UNIX code. Since
                there is not enough documentation for the W32 components of the system and
                also such documentation is usually misleading it can never be considered as
                100% emulation. Even in the final releases all the sanity checks
                implemented in this project should remain active as all the project's code
                always interacts with unknown and untrusted W32 binaries.</p>

                <p><span class="productname">Microsoft Windows NT</span> code is written in
                a&nbsp;foolproof style as it accepts even invalid input values, and which
                it usually corrects. This makes long-term debugging a&nbsp;pain as it hides
                sources of problems. &quot;Checked build&quot; releases were probably
                designed to fix this flaw by strict consistency checks but it did not reach
                its goals as such checks are usually missing in the code.</p>

                <p>This project has strict consistency checks across all the code to make
                the debugging phase easy enough. Failed sanity check is not always
                a&nbsp;bug &ndash; sometimes it just means the real W32 binary code is more
                benevolent than it could be expected according to the documentation and
                such sanity check gets removed for the next version build. In other cases
                the failed sanity checks mean the execution path for some unexpected
                arguments combination was not yet implemented by this project. I may also
                mean a bug, of course...</p>

                <p>Last but not least &ndash; never miss a&nbsp;possible sanity check as its
                later removal is in an order of magnitude cheaper than an&nbsp;uncaught
                invalid assumption. Failed assertion is not always a&nbsp;bug although it
                has to be fixed, of course.</p>


        <h2 id="logfile">STATUS_LOG_FILE_FULL</h2>

                <p>After writing approx. 1MB of data on NTFS test partition NTFS driver
                returns for any further write requests
                <span class="constant">STATUS_LOG_FILE_FULL</span> error code.
                Apparently it is caused by the fact this project is
                @{[ a_href 'Details.pm#synchronous','single-threaded' ]} and it ignores the spawn
                of parallel journalling thread during <span class="fname">ntfs.sys</span>
                initialization.</p>

                <p>Fortunately <span class="fname">ntfs.sys</span> will clear its
                journalling log file during filesystem unmount. This project will therefore
                remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</span>
                is detected to workaround missing journalling thread.</p>

                <p>Similiar behaviour can be seen during write of compressed files &mdash;
                the file gets written uncompressed and its compression will proceed only
                during the final filesystem unmount.</p>

                <p>For these reasons it was mandatory to support
                @{[ a_href 'Details.pm#parent_connector','transparent volume remounting' ]}.</p>


        <h2 id="parent_connector"><span class="constant">ParentConnector</span> volume remounter</h2>

                <p>The sandbox master component of this project has control of restarting
                its sandbox slaves containing the W32 filesystem. Target goal of
                <span class="constant">ParentConnector</span> component is to transparently
                provide persistent view of files and directories over the sandboxed slaves
                being restarted.</p>
                
                <p>In the case of read-only operations it would be simple as we could only
                save our state of currently opened filesystem objects with their read
                file/directory offset. Write operations can be handled as the read-only
                ones as long as all the operations are successful. In the case of W32
                filesystem crash we loose all the past write operations. If we would redo
                all the write operations we could very easily invoke the same crash.
                Therefore we write:</p>

                        <blockquote class="command">
                                <p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
                        </blockquote>

                <p>message to syslog and refuse any further operations with this
                object.</p>

                @{[ doc_img 'dia/parent-connector','Parent Connector' ]}

                <p><span class="constant">HANDLE</span> represents W32 object open in
                existing W32 filesystem.<span class="constant">HANDLE</span> is created
                on-demand according to the saved state of the object (such as its
                pathname). Even the whole <span class="constant">VFS</span> sandbox slave
                is spawn on-demand if some object operation requests it.</p>

                <p>W32 filesystem crash can obviously occur at any moment - it generates
                @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
                @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
                <span class="constant">abort</span>. Successful filesystem unmount
                (even as the part of remount operation) must be first preceded by
                <span class="constant">detach</span> signal to close all existing
                W32 <span class="constant">HANDLE</span>s. After their close the filesystem
                gets the unmount requests. Only in the case all the close operations
                succeeded including the final filesystem unmount the signal
                <span class="constant">cease</span> can be activated to notify all the
                dirty (written) objects they are now clean. During this
                <span class="constant">cease</span> signal the project will also
                @{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
                underlying media.</p>

                <p>Objects never written remain in <span class="constant">clean</span>
                state and they can be transparently reopened even if W32 filesystem crash
                occurs.</p>


HERE


exit;
}
1;