--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc APITypes page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::APITypes;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: API Functions',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<a name="functype"><h1>API Function Implementation Choices</h1></a>
+
+ <p>For each function exported by W32
+ <span class="fname">ntoskrnl.exe</span> and imported and called by the
+ filesystem driver a decision needs to be made to properly implement its
+ functionality. Currently implemented functionality statistics are provided
+ below:</p>
+
+ <table border="1" align="center">
+ <tr><th>Function type </th><th>Items</th><th>Portion</th></tr>
+ <tr><td>@{[ a_href '#functype_pass','pass' ]} </td><td> 81</td><td> 26%</td></tr>
+ <tr><td>@{[ a_href '#functype_wrap','wrap' ]} </td><td> 2</td><td> 0%</td></tr>
+ <tr><td>@{[ a_href '#functype_native_reactos','native-ReactOS' ]}</td><td> 113</td><td> 36%</td></tr>
+ <tr><td>@{[ a_href '#functype_native_libcaptive','native-own' ]} </td><td> 116</td><td> 38%</td></tr>
+ <caption>Function Implementation Types Statistics</caption>
+ </table>
+
+ <p>As there are several choices to implement each function the usual
+ attempts/investigations ordering is listed in the sections below.</p>
+
+ <p>Special case must be taken for data-type symbols since they are
+ referenced without the possibility of catching the code flow by some
+ breakpoints (it would be possible only in some special access cases). Data
+ export symbols of <span class="constant">unpatched</span> libraries must
+ contain already prepared content at the runtime. There is a problem
+ with <span class="constant">patched</span> libraries where it is necessary
+ to also fully implement the data symbol as
+ @{[ a_href '#functype_native','native implementation' ]} since there is no
+ possibility to @{[ a_href '#functype_pass','pass' ]} the data symbol instead of
+ the original W32 data location and therefore there will be two instances of
+ such data variable place. As there will be also the uncaught references for
+ such W32 data location from the <span class="constant">patched</span>
+ library itself such symbols should be usually only some constants (such as
+ <span class="constant">KeNumberProcessors</span>).</p>
+
+ <p>W32 platform symbols export/import can be based either on the symbol
+ name itself or it can be also exported and imported just by its
+ identification number called <span class="constant">Ordinal</span>.
+ Although it saves some jumptables file binary size it is currently no
+ longer used by W32 binaries and this project also does not support such
+ <span class="constant">Ordinal</span> symbol reference type at all.</p>
+
+ <p>All the exporting magic is handled by custom script
+ <span class="fname">captivesym</span> processing the definition file
+ <span class="fname">@{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym' ]}</span>
+ to produce the intermediate relaying code
+ <span class="fname">src/libcaptive/ke/exports.c</span>. For details of the
+ <span class="fname">captivesym</span>-specific source file syntax please
+ see its documentation:
+ <span class="fname">@{[ a_href
+ $W->{"top_dir"}.'/project/Pod2Html.html.pl?cvs=priv/captive/src/libcaptive/ke/captivesym.pl',
+ 'src/libcaptive/ke/captivesym.pl' ]}</span>
+
+ <a name="functype_pass"><h2>Direct Pass to Original "ntoskrnl.exe"</h2></a>
+
+ <p>Simple (standalone) functions such as
+ <span class="function">RtlTimeToSecondsSince1970()</span> can be simply
+ passed to the original implementation in
+ <span class="fname">ntoskrnl.exe</span> as they make no hardware access
+ and they do not expect any special internal data structures to be set up
+ in advance by an earlier library initialization. A common case are all
+ the data structures utility functions such as
+ <span class="constant">GenericTable</span> subsystem or
+ <span class="constant">LargeMcb</span> handling.</p>
+
+ <a name="functype_pass_fromunix"><h3>Pass from UNIX Code</h3></a>
+
+ <p>Control flow begins in some standard UNIX code. Such code is always
+ using @{[ a_href '#calltype_cdecl','cdecl call type' ]} for all its
+ intracalls. <a href="#functype_native_reactos">Native functions
+ compiled from <span class="productname">ReactOS</span> sources</a> use
+ their own @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} declarations
+ but these call type modifications are discarded during compilation for
+ this project by the <span class="constant">LIBCAPTIVE</span>
+ symbol.</p>
+
+ <p>UNIX code calls <span class="function">FUNCTIONNAME()</span> relay
+ from the generated UNIX jump table. Such relay will debug dump the
+ passed arguments and finally pass the control to the original W32
+ function code in the proper call type
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} for a given
+ function.</p>
+
+ <p>Original W32 code entry point is always trapped by a breakpoint
+ although it would not be needed during this specific direct pass from
+ UNIX code to the original W32 implementation. Still the breakpoint has
+ to be there to catch some other (such as intra-W32) possible calls
+ described later. There are several more ways to define breakpoint in
+ the code. One way is to use processor hardware breakpoint support but
+ the number of breakpoints is limited. The other way is to patch in the
+ <span class="instruction">@{[ 'int $3' ]}</span> instruction but it will invoke
+ <span class="constant">SIGTRAP</span> signal handler conflicting with
+ the possible debugger (<span class="productname">gdb(1)</span>)
+ control. This project uses the <span class="instruction">hlt</span>
+ instruction, which also has a single-byte opcode as
+ <span class="instruction">@{[ 'int $3' ]}</span> and it is a privileged
+ instruction forbidden to be used from the UNIX user space code.
+ <span class="instruction">hlt</span> invokes
+ <span class="constant">SIGSEGV</span> signal which can be resolved by
+ a custom signal handler without any conflict with the possible
+ debugger control; <span class="productname">gdb(1)</span> needs the
+ following command to pass through such
+ <span class="constant">SIGSEGV</span> signal:</p>
+
+ <blockquote class="command">
+ <p>handle SIGSEGV nostop noprint pass</p>
+ </blockquote>
+
+ <p>When a breakpoint gets caught, we usually need to return to the
+ running code. Unfortunately it is not possible because of the patched
+ breakpoint opcode. The breakpoint cannot be simply removed upon return
+ as it would permanently loose control over the point of entry. Even if
+ the return would include faking of the return address in the bottom
+ stack frame to patch the breakpoint back during later function exit it
+ still would not solve the caughts of inner calls of recursive
+ functions. One of the working possibilities would be to patch the
+ original instruction back and perform a singlestep provided by
+ <span class="function">ptrace(2)</span> syscall. However such
+ singlestep needs another controlling UNIX process and it would again
+ conflict with the debuggers such as
+ <span class="productname">gdb(1)</span>. This project implements the
+ singlestep functionality by two consecutive breakpoints
+ (<span class="instruction">hlt</span> instructions to be specific):
+ The first two instruction addresses of the W32 functions are called
+ <span class="productname">slot #1</span> and
+ <span class="productname">slot #2</span>, the length of the first
+ function instruction has to be analyzed to get the right address of
+ <span class="productname">slot #2</span>. When the first breakpoint is
+ caught it is necessary to patch the original instruction back and also
+ patch another breakpoint in place of
+ <span class="productname">slot #2</span>.
+ During the <span class="productname">slot #2</span> breakpoint
+ invocation the operation will be reverted — the breakpoint will be put
+ to <span class="productname">slot #1</span> again and the instruction
+ of <span class="productname">slot #2</span> will be restored to be able
+ to continue the execution of the function.</p>
+
+ <p>W32 function will finish in its specific
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}, the control
+ will return to the UNIX jump table relay which will debug dump the
+ return value and it will finally pass the control back to the UNIX
+ caller in the standard UNIX
+ @{[ a_href '#calltype_cdecl','cdecl call type' ]}.</p>
+
+ @{[ doc_img 'fig/functype_patched_pass_fromunix',
+ 'Function Type: <span class="constant">pass</span> from UNIX Code' ]}
+
+ <a name="functype_pass_fromw32"><h3>Pass from W32 Code</h3></a>
+
+ <p>This function type is similiar to the
+ @{[ a_href '#functype_pass_fromunix','previous one' ]} with the exception
+ of more complicated entry point. Unfortunately W32 libraries call their
+ own functions directly, using the <span class="instruction">call</span>
+ instructions without any patchable jump table. Even the
+ <span class="instruction">call</span> argument itself cannot be patched
+ according to the relocation table record as such library intra-call
+ instruction has no relocation due to its relative argument offset on
+ <span class="constant">i386</span>. This time the double-breakpoint
+ mechanism @{[ a_href '#functype_pass_fromunix','described above' ]} gets
+ handy since it will catch the entry point when the function gets
+ called. <span class="constant">SIGSEGV</span> handler gets invoked by
+ the <span class="instruction">hlt</span> instruction and it will
+ redirect the control to the jump table relay function to debug dump the
+ function entry arguments (it has no other uses in this call type).</p>
+
+ <p>When the relay needs to call the original function it will reach
+ exactly the same breakpoint instruction as during the recent
+ <span class="constant">SIGSEGV</span> handling redirecting to this
+ calling relay. But this time the
+ <span class="constant">through_w32_func</span> field of this function
+ record will be set to to prevent repeated redirection and to pass the
+ control through the breakpoint mangle instead this time.</p>
+
+ <p>Returning is not much interesting as the first
+ <span class="constant">SIGSEGV</span> handler did a straight jump
+ for the redirection purposes without any needed consequent
+ handling.</p>
+
+ <p>The jump table relay used for the callers from W32 code is
+ a different one than the relay being used for the callers
+ @{[ a_href '#functype_pass_fromunix','from UNIX code' ]}. UNIX code always
+ uses relay with external @{[ a_href '#calltype_cdecl','cdecl call type' ]}
+ but in this case a relay with the appropriate
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} is used.</p>
+
+ @{[ doc_img 'fig/functype_patched_pass_fromw32',
+ 'Function Type: <span class="constant">pass</span> from W32 Code' ]}
+
+ @{[ vskip() ]}
+
+ <table border="1" align="center">
+ <tr><td><span class="fname">captivesym</span> keyword</td><td>pass</td></tr>
+ <tr><td>Native code function name </td><td>(no implementation)</td></tr>
+ <tr><td>W32 traced code from UNIX function name </td><td>FUNCNAME</td></tr>
+ <tr><td>W32 traced code from W32 function name </td><td>FUNCNAME_cdecl/_stdcall/_fastcall</td></tr>
+ <tr><td>Entry/exit debug tracing from UNIX code </td><td>yes</td></tr>
+ <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
+ <caption>Function Type <span class="constant">pass</span> Characteristics</caption>
+ </table>
+
+ <a name="functype_wrap"><h2>Wrap of the Original "ntoskrnl.exe" Function</h2></a>
+
+ <a name="functype_wrap_fromunix"><h3>Wrapping of Call from UNIX Code</h3></a>
+
+ <p>The code control flow has no special hardcore features since it is
+ very similiar to <a href="#functype_pass_fromunix">the direct pass to
+ W32 function from UNIX code</a>. All the wrapping is done in the
+ standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.
+ Jump table debug dumping relays are provided twice — the
+ "outer" one to trace the parameters from the function caller
+ and the "inner" one to trace the call from the wrapper to the
+ original W32 code. The "inner" relay also calls the W32 code
+ with the appropriate <a href="#calltype">cdecl/stdcall/fastcall call
+ type</a>.</p>
+
+ @{[ doc_img 'fig/functype_patched_wrap_fromunix',
+ 'Function Type: <span class="constant">wrap</span> from UNIX Code' ]}
+
+ <a name="functype_wrap_fromw32"><h3>Wrapping of Call from W32 Code</h3></a>
+
+ <p>This scheme is a combination of the
+ <a href="#functype_wrap_fromunix">previous wrap of a call from
+ UNIX code</a> and the <a href="#functype_pass_fromw32">direct pass from
+ the W32 code</a>. The control is caught and redirected by
+ <span class="constant">SIGSEGV</span> handler from the breakpoint
+ placed at the entry to the original W32 function code. The second entry
+ to the original W32 function with the
+ <span class="constant">through_w32_func</span> field of this function
+ description already set is done from the "inner" jump table
+ relay with the appropriate
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}.</p>
+
+ @{[ doc_img 'fig/functype_patched_wrap_fromw32',
+ 'Function Type: <span class="constant">wrap</span> from W32 Code' ]}
+
+ @{[ vskip() ]}
+
+ <p>Some functions can be <a href="#functype_pass">passed to the original
+ code</a> but they need their parameters to be checked/prepared.
+ Currently, such wrapping is only needed for the
+ <span class="function">ExAllocateFromPagedLookasideList()</span> function
+ where it is required due to <a href="#init_ntoskrnl">missing execution of
+ <span class="fname">ntoskrnl.exe</span> initialization execution</a>,
+ which would otherwise properly initialize some internal data structures.
+ In this case the wrapping code detects passing of an uninitialized
+ parameter and will search through the whole
+ <span class="fname">ntoskrnl.exe</span> code body at runtime to find the
+ proper initialization routine containing the correct initialization
+ parameters. Passed addresses of static structures must be differentiated
+ as each of them usually has different initialization parameters. It is
+ proactive to not to have fixed parameters array as these parameters may
+ differ across different <span class="fname">ntoskrnl.exe</span>
+ versions.</p>
+
+ <table border="1" align="center">
+ <tr><td><span class="fname">captivesym</span> keyword</td><td>wrap</td></tr>
+ <tr><td>Native UNIX wrapping code function name </td><td>FUNCNAME_wrap</td></tr>
+ <tr><td>W32 traced wraping code from UNIX func. name </td><td>FUNCNAME</td></tr>
+ <tr><td>W32 traced wrapping code from W32 func. name </td><td>FUNCNAME_cdecl/_stdcall/...</td></tr>
+ <tr><td>W32 traced original code function name </td><td>FUNCNAME_orig</td></tr>
+ <tr><td>Entry/exit debug tracing from UNIX code </td><td>yes</td></tr>
+ <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
+ <caption>Function Type <span class="constant">wrap</span> Characteristics</caption>
+ </table>
+
+ <a name="functype_native"><h2>Native Implementation</h2></a>
+
+ <a name="functype_native_fromunix"><h3>Native Implementation Called from UNIX Code</h3></a>
+
+ <p>This is the simplest case of a function call as it is fully
+ handled only by the compiler and/or linker.</p>
+
+ <p>In this case though, no debug dumping call relay is provided — such
+ relay would need to rename the implementations of native functions to
+ prevent its automatic linking with the caller code. This renaming would
+ not be possible to do by simple <span class="constant">#define</span>
+ since it would also rename any calling statements of such function in
+ the same C sources. One of the possibilities to solve would be to
+ utilize <span class="dashdash">--redefine-sym</span> feature of the
+ <span class="productname">objcopy(1)</span> utility. On the other hand
+ there is not much need to catch/debug such calls as both the caller and
+ the callee are provided with full source file debug information for the
+ debugger. Also the callee usually debug dumps its entry/exit parameters
+ by custom debug dumps in the
+ <a href="#functype_native_reactos"><span class="productname">ReactOS</span> implementations</a>.
+
+ @{[ doc_img 'fig/functype_native_fromunix',
+ 'Function Type: <span class="constant">native</span> from UNIX Code' ]}
+
+ <a name="functype_native_fromw32"><h3>Native Implementation of
+ "unpatched" Library Function Called from W32 Code</h3></a>
+
+ @{[ doc_img 'fig/functype_unpatched_native_fromw32',
+ 'Function Type: <span class="constant">native</span> of <span class="constant">unpatched</span> from W32 Code' ]}
+
+ <p>Here comes the differentiation if the project deals either with
+ a <span class="constant">patched</span> or an
+ <span class="constant">unpatched</span> version of the library
+ (<span class="constant">patched</span> is a loaded W32 binary
+ library while <span class="constant">unpatched</span> library is
+ completely provided by this project with no use of the library's
+ original W32 binary file). As the project adjusts the exported symbol
+ address during the patching operation, in some cases the
+ <span class="constant">patched</span> library call may be handled
+ simply as <span class="constant">unpatched</span> library call even for
+ the <span class="constant">patched</span> libraries. Fortunately the
+ distinction is not much important as the project is prepared to
+ properly handle both cases.</p>
+
+ <p>The W32 caller which imported the symbol will be pointed right to
+ the relaying function. The debug dumping relay will be called from W32
+ code with the appropriate
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} while the
+ relay will call the implementation of the native function in the
+ standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.</p>
+
+ <a name="functype_native_fromw32_patched"><h3>Native Implementation of "patched" Library Function Called from W32 Code</h3></a>
+
+ @{[ doc_img 'fig/functype_patched_native_fromw32',
+ 'Function Type: <span class="constant">native</span> of <span class="constant">patched</span> from W32 Code' ]}
+
+ <p>The calling scheme is similiar to the
+ <a href="#functype_native_fromw32">previous call of
+ <span class="constant">unpatched</span> library function from W32
+ code</a> but the call control is redirected from the entry point of the
+ original W32 binary implementation by the breakpoint and its
+ <span class="constant">SIGSEGV</span> handler as in
+ <a href="#functype_pass_fromw32">the case of passing control from W32
+ call</a>.</p>
+
+ <p>The original W32 function implementation located in the original
+ loaded binary file is never executed but its entry point needs to be
+ trapped by the breakpoint to be able to catch the function calls within
+ the library.</p>
+
+ @{[ vskip() ]}
+
+ <p>In all cases the final function implementation is a standard UNIX
+ code compiled from C sources with full debug information available
+ for the debugger. Fortunately all such functions do not need to be coded
+ from scratch for this project since there already exist $freespeech
+ $ReactOS and $Wine projects and their code can be used instead.</p>
+
+ <p>$Wine project is listed mostly for a completeness as almost no
+ code was suitable for reuse as it implements W32 user space while this
+ project is running pure W32 kernel space environment (in $gnulinux user
+ space!).</p>
+
+ <a name="functype_native_reactos"><h3>Native Implementation
+ - <span class="productname">ReactOS</span></h3></a>
+
+ <p>Some functions are already implemented in the $ReactOS
+ project and they can be used as they are. Although it would be
+ possible to <a href="#functype_pass">pass some function calls to the
+ original code</a> it is more handy to provide native implementation as
+ there is better control of the data handling during debugging sessions
+ due to the provided debugging symbols.</p>
+
+ <p>Such functions can be found in
+ <span class="fname">src/libcaptive/reactos/</span> subdirectory.
+ Some functions had to be adjusted for this project
+ - these modifications are compiled conditionally, depending on the
+ <span class="constant">LIBCAPTIVE</span> symbol existence.</p>
+
+ <p>Later stages of this project reached the level where
+ $ReactOS is yet too immature and the needed functions are usually
+ written just with the sad body:</p>
+
+ <blockquote class="command">
+ <p>UNIMPLEMENTED;</p>
+ </blockquote>
+
+ <p>Functions that were not possible to
+ @{[ a_href '#functype_pass','pass' ]} were reimplemented by this project
+ and placed in the project's implementation directories
+ @{[ a_href '#reactos_nocare','instead of extending' ]} $ReactOS code.</p>
+
+ <a name="functype_native_wine"><h3>Native Implementation – <span class="productname">Wine</span></h3></a>
+
+ <p>Even though $Wine only implements the
+ <span class="productname">Microsoft Windows NT</span> user space, there
+ still are some common functions which could be copied from the $Wine
+ project.</p>
+
+ <a name="functype_native_libcaptive"><h3>Native Implementation – Project Specific</h3></a>
+
+ <p>As the last resort it was necessary to provide completely own
+ implementation of some API functions such as PC hardware dependent
+ parts or memory management functions.</p>
+
+ @{[ vskip() ]}
+
+ <table border="1" align="center">
+ <tr><td><span class="fname">captivesym</span> keyword</td><td>(none; just the symbol name)</td></tr>
+ <tr><td>Native code function name </td><td>FUNCTIONNAME</td></tr>
+ <tr><td>Native traced code from W32 code func. name </td><td>FUNCTIONNAME_cdecl/_std...</td></tr>
+ <tr><td>Entry/exit debug tracing from UNIX code </td><td>no</td></tr>
+ <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
+ <caption>Function Type <span class="constant">native</span> Characteristics</caption>
+ </table>
+
+ <a name="functype_undef"><h2>Undefined Function</h2></a>
+
+ <p>Functions not defined by any of the previous function types cannot be
+ called by any W32 code including the code of the library implementing
+ such function. All functions of <span class="constant">patch</span>ed
+ libraries not listed in the <span class="fname">captivesym</span> exports
+ file are automatically set to be trapped as fatal program execution
+ errors.</p>
+
+ <p>It is not necessary to list the symbols as
+ <span class="constant">undef</span> as long as you are just loading the
+ W32 <span class="constant">PE-32</span> code and the symbols belong to
+ <span class="constant">patch</span>ed library. On the other hand if you
+ are loading W32 <span class="fname">.so</span> code or if such symbol is
+ a part of <span class="constant">unpatched</span> library (and thus
+ being completely provided by the project) you need to list such symbol as
+ <span class="constant">undef</span> type to prevent unresolved symbol
+ reference.</p>
+
+ <table border="1" align="center">
+ <tr><td><span class="fname">captivesym</span> keyword</td><td>undef</td></tr>
+ <tr><td>Native code function name </td><td>(no implementation)</td></tr>
+ <tr><td>Native traced code function name </td><td>FUNCTIONNAME_cdecl/_stdcall/_fastcall</td></tr>
+ <tr><td>Debug tracing message from UNIX code </td><td>yes</td></tr>
+ <tr><td>Debug tracing message from W32 code </td><td>yes</td></tr>
+ <caption>Function Type <span class="constant">undef</span> Characteristics</caption>
+ </table>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc About page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::About;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: About',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<h1>About</h1>
+
+ <a name="reasons"><h2>Reasons for the Implementation</h2></a>
+
+ <p>Currently there is no possibility to any of the available $freespeech
+ ($freespeech used in the following text in the meaning of
+ "@{[ a_href 'http://www.gnu.org/philosophy/free-sw.html','free as in speech' ]}")
+ operating systems to reliably write to the most common disk partition
+ filesystem type – <span class="productname">Microsoft NTFS</span>. It would
+ be already supported a long time ago but there is no proper documentation of
+ <span class="productname">NTFS</span> filesystem data structures available.
+ Since <span class="productname">Microsoft</span> corporation continues in its
+ propagation of <span class="productname">Microsoft Windows NT</span>
+ (<span class="productname">NT</span> identifier used in the following text
+ applies to all the products of <span class="productname">Microsoft</span>
+ <span class="productname">NT</span> series such as
+ <span class="productname">NT 4.0</span>,
+ <span class="productname">2000</span> as NT-5.0
+ and
+ <span class="productname">XP</span> as NT-5.1.)
+ based operating systems <span class="productname">NTFS</span> is the default
+ disk file system type for vendor preinstalled <span class="productname">Microsoft Windows</span>.
+
+ <p>Unfortunately the <span class="productname">NTFS</span> filesystem has too
+ complex data structure to allow a complete reverse enginnering process in
+ reasonable time. Currently available $freespeech solutions such as $LinuxNTFS
+ filesystem have already implemented reliable reverse
+ engineered read-only access. However <a name="reliability">reliabile</a>
+ read-write part of the access would require much better
+ knowledge of the <span class="productname">NTFS</span> data structures.
+ Currently only rewriting of already existing file data blocks is supported
+ by $LinuxNTFS — no file creation, no file deletion, no directory operations etc.
+ Also any future versions of <span class="productname">NTFS</span> filesystem
+ would require another major reverse engineering effort.</p>
+
+
+ <a name="challenges"><h2>Challenges of the Project</h2></a>
+
+ <p>The <a name="NTFSgoal">ultimate goal</a> of this project is definitely the
+ free implementation of @{[ a_href '#reliability','reliable' ]} read-write <span
+ class="productname">NTFS</span> filesystem driver. This project chose to
+ solve this problem in the style of $Wine project by using the original binary
+ <span class="fname">ntfs.sys</span> and emulating all the required layers of
+ <span class="productname">Microsoft Windows NT</span> for it.</p>
+
+ <p>Unfortunately this effort is tainted by only partial and generally
+ insufficient documentation of API between filesystem driver
+ (<span class="fname">ntfs.sys</span>) and the
+ <span class="productname">Microsoft Windows NT</span>
+ ("@{[ a_href 'http://mail.gnu.org/archive/html/libtool/2000-09/msg00000.html','W32' ]}"
+ in the following text) kernel <span class="fname">ntoskrnl.exe</span>. Note
+ that this API is a different than the one being used in the $Wine project
+ since <span class="productname">Wine</span> implements only the user space
+ part of W32.</p>
+
+
+ <a name="versions"><h2>Microsoft Windows Versions Compatibility</h2></a>
+
+ <p>Currently this project supports only driver files of
+ <span class="productname">Microsoft Windows XP</span> (NT-5.1)
+ in the following releases:</p>
+
+ <ul>
+ <li>Microsoft Windows XP Service Pack 1a Checked Build U.S.</li>
+ <li>Microsoft Windows XP Service Pack 1a Free Build U.S.</li>
+ <li>Microsoft Windows XP (No Service Pack) Checked Build U.S.</li>
+ <li>Microsoft Windows XP (No Service Pack) Free Build U.S.</li>
+ </ul>
+
+ <p>The latest list of supported driver files can be found in:
+ @{[ captive_srcfile 'src/install/acquire/w32-mod-id.captivemodid.xml' ]}</p>
+
+ <p>Microsoft Windows NTFS filesystem driver is capable of accessing even
+ the older formats of the filesystem. Porting to Microsoft Windows 2003
+ Server is expected to be done soon. There can be also a danger of Microsoft
+ Windows upgrading NTFS disk filesystem where you would no longer be able
+ to access your NTFS disk by your original
+ <span class="productname">Microsoft Windows</span> version.
+ This upgrade does not happen as it occurs only during complete CD-ROM
+ Microsoft Windows system installation – such operation is not performed
+ by this project.</p>
+
+ <p>There is no problem technically obtaining the needed driver files of
+ <span class="productname">Microsoft Windows XP</span> as they are freely
+ downloadable at:
+ @{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp' ]}</p>
+ There may be @{[ a_href '#law','legal reasons' ]} you would not be allowed
+ to use there files if you own license to a different version of
+ <span class="productname">Microsoft Windows</span>. Legal rights will very
+ depending on your country.</p>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Architecture page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::Architecture;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: Architecture',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<h1>Architecture</h1>
+
+ <p>The principle of the
+ project lies in the glue between
+ <span class="productname">Microsoft Windows NT</span> kernel space
+ environment and $gnulinux user space process environment:</p>
+
+ @{[ doc_img 'arch-W32','Microsoft Windows Subsystems Architecture' ]}
+ @{[ doc_img 'arch-captive','Captive Subsystems Architecture' ]}
+
+ <a name="existing_emulation"><h2>Existing Emulation Projects</h2></a>
+
+ <p>There were two well-known $freespeech projects emulating W32 subsystems
+ to reach the compatibility with various W32 components:
+ $Wine and $ReactOS. Sad moment is that the goals of this project do not fit
+ very well into any role in those two ones. Therefore this project went
+ its own way of emulation:</p>
+
+ <table align="center" border="1">
+ <tr>
+ <th>@{[ a_href '#guestosnote','Guest-OS' ]}</th>
+ <th>@{[ a_href '#hostosnote' ,'Host-OS' ]}</th>
+ <th>Implements</th>
+ <th>W32 kernel library</th>
+ </tr>
+ <tr>
+ <td>$Wine</td>
+ <td>$gnulinux</td>
+ <td>W32 user space</td>
+ <td><span class="fname">ntdll.dll</span></td>
+ </tr>
+ <tr>
+ <td>$ReactOS</td>
+ <td><span class="constant">i386</span> hardware</td>
+ <td>W32 kernel and user space</td>
+ <td><span class="fname">ntoskrnl.exe</span></td>
+ </tr>
+ <tr style="height: 1ex;"></tr>
+ <tr>
+ <td>this project</td>
+ <td>$gnulinux</td>
+ <td>W32 kernel</td>
+ <td><span class="fname">ntoskrnl.exe</span></td>
+ </tr>
+ <caption>Emulation Projects Characteristics</caption>
+ </table>
+
+ <dl>
+ <a name="guestosnote"><dt>Guest-OS</dt></a>
+ <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#guestos','Guest OS' ]}:
+ An operating system that runs inside a virtual machine.</dd>
+ <a name="hostosnote" ><dt>Host OS</dt></a>
+ <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#hostos' ,'Host OS' ]}:
+ An operating system that runs on the host machine.</dd>
+ </dl>
+
+ <p>While $ReactOS provides the necessary W32 kernel subsystem emulation
+ code we also need to run such @{[ a_href '#guestosnote','Guest-OS' ]} in the
+ @{[ a_href '#hostosnote','Host-OS' ]} $gnulinux. Initially it was planned to
+ extend $Wine with the W32 kernel space emulation functionality but
+ fortunately <span class="author">Steven Edwards</span> pointed to the $ReactOS
+ which better suits the needs of this project by its already implemented W32
+ kernel space emulation.</p>
+
+ <p>The <a name="reactos_nocare">original reasons</a> for developing
+ $ReactOS still make no sense to the author of this project. Free
+ implementation of W32 platform standalone running on the machine hardware
+ is no longer free as most od the W32 applications are usually closed source
+ and the user still looses its freedom on the application level anyway. Even
+ in the case of available free applications there still remains the
+ disadvantage of loosing the Host-OS platform availability if implemented in
+ the $Wine style. For these ideology incompatibilities not much effort was
+ made for acceptance the fixes and improvements of $ReactOS by this project.
+ Moreover new functionality is not being implemented to the $ReactOS part
+ but it is coded in Gnome style in the project specific source files
+ place.</p>
+
+ <p>The most serious problem of $ReactOS is its dependence on the direct
+ <span class="constant">i386</span> hardware instead of some
+ @{[ a_href '#hostosnote','Host-OS' ]} as required by the goals of this project.
+ W32 is designed to be hardware-independent using its
+ <span class="fname">hal.dll</span>. Unfortunately $ReactOS does not follow
+ this design and thus there are needed various patches and replaces of its
+ various parts and its hardware-dependent code. Despite it $ReactOS code
+ base still made a big asset for this project.</p>
+
+
+
+
+
+ <p>Some API functions are provided both by
+ <span class="fname">ntdll.dll</span> and
+ <span class="fname">ntoskrnl.exe</span> in W32.
+ <span class="author">Casper Hornstrup</span> enlightened such functions
+ calling conventions have to be differentiated as
+ <span class="fname">ntdll.dll</span> lives in the user space (low address
+ space – below <span class="constant">0x80000000</span>) and
+ <span class="fname">ntoskrnl.exe</span> in the kernel space (high address
+ space – above <span class="constant">0x80000000</span>). Although they
+ contain slightly different set of symbols (functions)
+ <span class="fname">ntdll.dll</span> still can be considered as a user
+ space interface to the kernel space implementation by
+ <span class="fname">ntoskrnl.exe</span>.</p>
+
+ <p>Currently there are
+ no plans to ever extend the project's crossplatformity beyond the
+ <span class="constant">i386</span> processor
+ (<span class="constant">i386</span> used here as
+ @{[ a_href 'http://www.intel.com/','Intel' ]} architecture covering 32-bit
+ processors compatible with <span class="constant">i386</span>,
+ <span class="constant">i486</span>, ...).</p>
+
+
+ <a name="law"><h2>Laws and Licensing Conditions</h2></a>
+
+ <p>If you are an <span class="productname">authorized user</span> of
+ <span class="productname">Microsoft Windows NT</span> the laws in some
+ countries give you the right to fully handle the product in any way you
+ want. Therefore you can disassemble the product even in the case you had
+ to agree with the product license forbidding such disassembly as the
+ country laws override any such license agreement.</p>
+
+ <a name="law_servicepack"><h3>Microsoft Service Pack</h3></a>
+
+ <p>Sometimes you may have the legal license for
+ <span class="productname">Microsoft Windows NT</span>
+ but for various technical reasons you do not have the media and/or
+ installation ready at the place of intended use of this project.</p>
+
+ <p>Fortunately <span class="productname">Microsoft</span> provides
+ $freebeer update packages for its
+ <span class="productname">Microsoft Windows</span> products called
+ <span class="productname">Service Packs</span>; the latest one is
+ <span class="productname">@{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp','Microsoft Windows XP Service Pack 1a' ]}</span>.</p>
+
+ <p>This downloadable file contains the full versions of the essential
+ files needed for the current stage of this product:
+ <span class="fname">ntfs.sys</span>
+ and
+ <span class="fname">ntoskrnl.exe</span>.
+ It even contains
+ <span class="fname">cdfs.sys</span> and
+ <span class="fname">fastfat.sys</span> for testing purposes.</p>
+
+ <p><span class="productname">Service Pack</span> also contains
+ EULA (End User License Agreement) paper disallowing any use of
+ <span class="productname">Service Pack</span> outside its original
+ intentions. According to the laws of some countries you need to be
+ <span class="productname">authorized user</span> of the
+ <span class="productname">Microsoft Windows XP</span> product to be
+ allowed to use the files contained in such
+ <span class="productname">Service Pack</span> without the bindings of its
+ EULA. Even the interpretation of such laws may vary.</p>
+
+ <p>It would be a breach of the law by the project author to provide
+ automatic (=hidden) functionality to download and extract the
+ <span class="productname">Service Pack</span> files. On the other hand it
+ is perfectly legal to ask user for his/her confirmation whether he/she is
+ really the <span class="productname">authorized user</span> of
+ <span class="productname">Microsoft Windows XP</span> product and
+ download/extract the <span class="productname">Service Pack</span> files
+ accordingly.</p>
+
+ @{[ doc_img 'captive-install-acquire-ask','Microsoft Windows Drivers Acquire Affirmation' ]}
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Cache Manager page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::CacheManager;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: NT Cache Manager',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<a name="cache_manager"><h1>NT Cache Manager</h1></a>
+
+ <p>Although there exist some 3rd party documents about
+ <span class="productname">NT Cache Manager</span> W32 subsystem such as
+ <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
+ 'The NT Cache Manager Description' ]}</span> or
+ <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
+ 'Learn About NT'."'".'s File-system Cache' ]}</span>
+ they are definitely insufficient for compatible
+ <span class="productname">NT Cache Manager</span> reimplementation.</p>
+
+ <p><span class="productname">NT Cache Manager</span> is about mapping
+ filesystem objects such as regular file data, filesystem bitmap or
+ journalling zone (log file). It is also being used by the filesystem for
+ mapping of virtual volume files representing the whole underlying
+ filesystem device.</p>
+
+ <p>The original W32 <span class="productname">NT Cache Manager</span>
+ is much more complicated as it must coordinate its effort with
+ other W32 subsystems like mapping of executable files
+ (<span class="type">ImageSectionObject</span>), insufficient system
+ resources from <span class="productname">NT Memory Manager</span>
+ or general effort to perform caching features for system performance.</p>
+ <span class="productname">NT Cache Manager</span> of this project has much
+ simpler goal - it just needs to provide compatible
+ <span class="productname">NT Cache Manager</span> functionality while
+ the other goals of its W32 counterpart are left to be successfuly handled
+ by UNIX OS in much more efficient way.</p>
+
+ @{[ doc_img 'dia/cache-manager',
+ '<span class="productname">NT Cache Manager</span> Architecture' ]}
+
+ <p>Cache Manager objects are always bound to
+ <span class="type">FCB</span> (File Control Block).
+ <span class="type">FileObject</span> (or its associated
+ <span class="type">HANDLE</span>) serve only as reference
+ to <span class="type">FCB</span> and there can be multiple
+ <span class="type">FileObject</span>/<span class="type">HANDLE</span>
+ items for one <span class="type">FCB</span>. It is a bit misleading
+ you must use <span class="type">FileObject</span> pointer while calling
+ most of the Cache Manager functions.</p>
+
+ <p>Before using any other Cache Manager functions you must first call
+ <span class="function">CcInitializeCacheMap()</span>. You must give the
+ maximum mapped object offset. Each mapped object byte must have at most one
+ mapped memory location - no shared pages are allowed. Also any subsequent
+ mapping request is expected to be mapped into continuous memory region.
+ It implies you must reserve the memory region for possible future mapping
+ during the initial <span class="function">CcInitializeCacheMap()</span>
+ moment sized according to the given maximum mapped object offset.
+ This is the approach currently implemented by this project although it
+ cannot be used for 3rd party <span class="fname">ext2fsd.sys</span>
+ driver as it initialized Cache Manager by the whole media device size
+ and it surprisingly succeeds for original
+ <span class="productname">Microsoft Windows</span>
+ <span class="productname">Cache Manager</span>.
+ I expect the space reservation should be postponed to the first mapping
+ request and expect no multiple mappings will be done in the case
+ of memory-exceeding <span class="function">CcInitializeCacheMap()</span>
+ reservation request. <span class="function">CcSetFileSizes()</span>
+ changing the reserved memory area size may assume no existing Map
+ or Pin mappings exist.</p>
+
+ <p><span class="type">PCACHE_MANAGER_CALLBACKS</type> argument can be
+ safely ignored:</p>
+
+ <dl>
+ <dt><span class="function">AcquireForReadAhead()</span>/<span class="function">ReleaseFromReadAhead()</span></dt>
+ <dd>
+ <p>As any readahead functionality is optional these entries are
+ never used by Cache Manager implementation of this project.</p>
+ </dd>
+
+ <dt><span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span></dt>
+ <dd>
+ <p>Even the write-behind functionality is optional for Cache Manager.
+ It is being done in asynchronous way in the original
+ <span class="productname">Microsoft Windows</span>
+ <span class="productname">Cache Manager</span>.
+ implementation and it is ignored by Cache Manager implementation of
+ this project.</p>
+
+ <p>Cache Manager does not need to write any data if not explicitely
+ requested by the driver. It is even expected to silently drop any
+ pending dirty data blocks during filesystem shutdown.
+ Forced dirty block write by function
+ <span class="function">CcFlushCache()</span> should be written without
+ any wrapping surrounding
+ <span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span>
+ pair.</p>
+ </dd>
+ </dl>
+
+ <p><span class="function">CcUninitializeCacheMap()</span> is just
+ a suggestion for Cache Manager that driver will no longer reference
+ given <span class="type">SharedCacheMap</span>. The uninitialization
+ can be postponed to any later moment in original
+ <span class="productname">Microsoft Windows</span>
+ <span class="productname">Cache Manager</span>
+ as it may be locked by existing
+ <span class="type">ImageSectionObject</span>
+ of some file being executed etc.
+ <a name="sharedcachemap_leak">It is fatal to destroy
+ <span class="type">SharedCacheMap</span></a>
+ in the moment you see no other
+ references to it as the driver will access it for some moment
+ even after <span class="function">CcUninitializeCacheMap()</span>.
+ I am not sure if it is a bug of the driver or whether there are some rules
+ how long after <span class="function">CcUninitializeCacheMap()</span>
+ completion given <span class="type">SharedCacheMap</span> still exists.
+ Fortunately it is safe to never destroy
+ <span class="type">SharedCacheMap</span> and leave it leaked - everything
+ gets clean in the
+ @{[ a_href '#sandbox','sandboxed environment' ]} soon anyway.</p>
+
+ <p>There exist Map and Pin type objects for each
+ <span class="type">SharedCacheMap</span> although they look very similiar.
+ Only these objects give you access to any memory data
+ — <span class="type">SharedCacheMap</span> only reserved the space
+ to ensure continuous mapping of the forthcoming mappings but it did not map
+ any data into it.</p>
+
+ <p>Mapping of 'new' Map or Pin will create the new object only in the case
+ no such mapping exists now. Otherwise you will just get the reference to
+ the existing object with increased usecount.</p>
+
+ <dl>
+ <dt>Map</dt>
+ <dd>
+ <p>Map mapping is always at most one for each
+ <span class="type">SharedCacheMap</span>. Base offset/length of such
+ mapping have no meaning as there can be only single Map.</p>
+
+ <p>Apparently Map size can be arbitrary long according
+ to its <span class="type">SharedCacheMap</span> reserved space.</p>
+
+ <p>You cannot modify the memory mapped by Map in any way.
+ As it is the same memory area (address) as the pages used by Pin
+ objects you always access the last modified version by possible
+ Pin of the same page.</p>
+ </dd>
+
+ <dt>Pin</dt>
+ <dd>
+ <p>Pin mapping always represents just one physical page
+ (<span class="constant">PAGE_SIZE</span> &nspan; 4096 for i386).
+ Its base offset/length can be safely extended to be aligned to the
+ requested page.</p>
+
+ <p>Pin can have associated pair of oldest and newest
+ <span class="type>LSN</span> (Linear Sequence Number). It can be
+ set by <span class="function">CcSetDirtyPinnedData()</span>
+ and Cache Manager always tracks the lowest and highest
+ reported <span class="type>LSN</span> for each page.
+ <span class="type>LSN</span> is assumed to be
+ <span class="constant">0</span> if not set.</p>
+
+ <p>Any existing Pin mapping will be reused for further mappings
+ as long as it is not ThreadOwned. In the moment you use
+ <span class="function">CcSetBcbOwnerPointer()</span> you will detach
+ the associated Pin pages from its
+ <span class="type">SharedCacheMap</span>.
+ Although they will further act as valid Pin mappings they will be no
+ longer reused during new Pin mapping of the same page.
+ There can exist multiple Pin mappings of the same page (although
+ sharing the same memory space). This detaching must be implemented
+ even in the
+ @{[ a_href '#synchronous','single-threaded' ]} W32 implementation
+ of this project as it is affecting the behaviour of Cache Manager.
+ It was never
+ @{[ a_href '#TraceFS','seen' ]} how to behave if multiple dirty Pin
+ mappings of the same page exist.</p>
+ </dd>
+ </dl>
+
+ <p>Only the pages not yet present in the memory must be read from the disk.
+ You must not read any pages you do not need to as the driver does not
+ expect it and it would corrupt its data buffers.</p>
+
+ <p>Cache Manager of this project will destroy Pin or Map mappings after
+ their last unreferencing (in opposite of
+ @{[ a_href 'sharedcachemap_leak','leaked <span class="type">SharedCacheMap</span>' ]}).
+ Despite it any dirty pages may still be held as the pages
+ (including their <span class="type>LSN</span>s) are cached associated
+ with <span class="type">SharedCacheMap</span>. It may be also possible
+ original <span class="productname">Microsoft Windows</span>
+ <span class="productname">Cache Manager</span>
+ postpones Pin mapping destroy to later time but it does not matter.</p>
+
+
+ <a name="TraceFS"><h2>TraceFS NT Cache Manager Tracer</h2></a>
+
+ <p>@{[ a_href '#cache_manager','Cache Manager behaviour' ]} would be hard
+ to analyze just by @{[ a_href '#reverse','reverse engineering' ]} as it
+ is pretty complicated code cooperating with many other W32 kernel
+ subsystems. It was chosen as easier way to trace it instead and validate
+ all the Cache Manager assumptions by Cache Manager simulator.</p>
+
+ @{[ doc_img 'dia/TraceFS','TraceFS Hooking' ]}
+
+ <p>You must prepare your driver to be hooked
+ (<span class="fname">ntfs.sys</span> in this case):</p>
+
+ <blockquote class="command">
+ <p>@{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} ntfs.sys ./src/TraceFS/TraceFS-W32/TraceFS.sys >hooked/ntfs.sys</p>
+ </blockquote>
+
+ <p>This <span class="fname">hooked/ntfs.sys</span> file must be replaced
+ in the <span class="fname">%System32%\\drivers</span> directory.
+ Beware as
+ <span class="productname">Microsoft Windows</span>
+ has many backups of these system files such as
+ <span class="fname">%System32%\\dllcache</span> — delete them
+ all!</p>
+
+ <p>You also need to install
+ <span class="fname">./src/TraceFS/TraceFS-W32/TraceFS.sys</span>
+ into <span class="fname">%System32%\\drivers</span> directory
+ and import <span class="fname">TraceFS/TraceFS-W32/TraceFS.reg</span>
+ registry file to initialize the debug driver during system boot.</p>
+
+ <p>You can now pray a bit and snap the resulting Cache Manager tracing
+ from <span class="productname">WinDbg</span> by
+ @{[ a_href '#WinDbg','W32 remote kernel debugging' ]}:</p>
+
+ @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}
+
+ <p>The resulting trace file should be processed by
+ @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
+ implementation to validate its assumptions about Cache Manager behaviour.
+ Any seen incompatibilies will be reported — your target is to reach
+ as few error messages as possible.</p>
+
+ <p>KNOWN BUGS: Combination of message synchronization primitives and
+ implemented refusal to create journalling thread of
+ <span class="fname">ntfs.sys</span>
+ causes fatal system lockup in several advanced operations
+ such as setting compression attribute. Despite it more common operations
+ can be successfuly traced during the whole
+ <span class="productname">Microsoft Windows</span>
+ session including its final shutdown and such traces provide enough
+ material to be food to
+ @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
+ validator.</p>
+
+ <a name="TraceFS_general"><h3>TraceFS for general API tracing</h3></a>
+
+ <p>Although TraceFS was up to now used only for tracing of
+ <span class="productname">NT Cache Manager</span> it can be easily
+ used ever for any other NT kernel API tracing. You need to provide
+ appropriate function wrappers in the main source file
+ @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.c' ]}
+ and you must also export them in
+ @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.def' ]}.
+ @{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} has no hardcoded
+ function names – it will hook exactly the exported entries.</p>
+
+ <p>Framework for thread synchronizations and debug tracing is provided to
+ prevent mangling of messages while running by multiple threads at once.
+ Testing was done just on uniprocessor machine, SMP kernel may need some
+ fixes.</p>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Calling Types page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::CallType;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: API Calling Conventions',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<a name="calltype"><h1>API Function Calling Conventions</h1></a>
+
+ <p>Standard UNIX code compiled by GCC (GNU C Compiler) running on host
+ $gnulinux always uses @{[ a_href '#calltype_cdecl','cdecl' ]} ABI (Application
+ Binary Interface) calling convention. This calling convention is also the
+ default declaration type of UNIX functions.</p>
+
+ <p>W32 uses three different calling conventions in its ABI. They are all
+ described in the
+ <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclang/html/_core_argument_passing_and_naming_conventions.asp"><span class="productname">Microsoft</span> documentation</a>.
+ There is always necessary to have the proper function declaration
+ (prototype) in the caller scope to prevent all sorts of unexpected
+ crashes.</p>
+
+ <p>Unfortunately some non-matching combinations of calling conventions
+ result in hard to debug bugs: the caller gets back an unexpected stack
+ pointer from the callee and upon return it will restore registers from the
+ wrong stack pointer place. Since the caller will finally reclaim its stack
+ frame from its (uncorrupted) <span class="constant">EBP</span> stack frame
+ pointer the caller will return to the caller of the caller correctly. Just
+ the registers remain corrupted causing crashes of completely unrelated code
+ executed far, far away...</p>
+
+ <p><span class="constant">EDI</span>, <span class="constant">ESI</span> and
+ <span class="constant">EBX</span> registers are always saved on the stack.
+ They are stored on the stack in this particular order from bottom to top
+ addresses (using the <span class="instruction">push EBX</span>,
+ <span class="instruction">push ESI</span>,
+ <span class="instruction">push EDI</span> sequence). Fortunately $gnulinux
+ GCC has the same register saving behaviour. If some register corruption
+ occurs the calling type presented between the caller and callee should be
+ checked.</p>
+
+ <a name="calltype_cdecl"><h2>W32 Calling Convention "cdecl"</h2></a>
+
+ <p>The only calling convention in the UNIX world. The default one for all
+ the compilers. All the arguments are passed on the stack, no arguments
+ are cleaned by the callee. Possible inconsistencies in the number of
+ function arguments with the function prototype used by the caller is
+ harmless. Variable arguments lists can be passed by this convention.</p>
+
+ @{[ doc_img 'fig/calltype_cdecl',
+ 'W32 Calling Convention <span class="constant">cdecl</span> Scheme' ]}
+
+ <table border="1" align="center">
+ <tr><td>Arguments freed by </td><td>caller</td></tr>
+ <tr><td>Arguments on the stack </td><td>#0 ... #(n-1)</td></tr>
+ <tr><td>Arguments in the registers </td><td>none</td></tr>
+ <tr><td>GCC attribute </td><td><span class="command">__attribute__((__cdecl__))</span> (default)</td></tr>
+ <caption>Calling Convention <span class="constant">cdecl</span> Characteristics</caption>
+ </table>
+
+ <a name="calltype_stdcall"><h2>W32 Calling Convention "stdcall"</h2></a>
+
+ @{[ doc_img 'fig/calltype_stdcall',
+ 'W32 Calling Convention <span class="constant">stdcall</span> Scheme' ]}
+
+ <p>Convention never used in the UNIX world. It needs to be specified for
+ W32 compilers. All the arguments are passed on the stack, all the
+ arguments are cleaned by the callee. Possible inconsistencies in the
+ number of function arguments with the function prototype used by the
+ caller will result in fatal crash. Variable arguments lists cannot be
+ passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
+ instead.</p>
+
+ <table border="1" align="center">
+ <tr><td>Arguments freed by </td><td>callee</td></tr>
+ <tr><td>Arguments on the stack </td><td>#0 ... #(n-1)</td></tr>
+ <tr><td>Arguments in the registers </td><td>none</td></tr>
+ <tr><td>GCC attribute </td><td><span class="command">__attribute__((__stdcall__))</span></td></tr>
+ <caption>Calling Convention <span class="constant">stdcall</span> Characteristics</caption>
+ </table>
+
+ <a name="calltype_fastcall"><h2>W32 Calling Convention "fastcall"</h2></a>
+
+ <p>Convention never used in the UNIX world. It needs to be specified for
+ W32 compilers. Convention used in the W32 world for its low calling
+ overhead. All but the first two arguments are passed on the stack, such
+ arguments are cleaned by the callee. First two arguments are passed in
+ the registers <span class="constant">ECX</span> and
+ <span class="constant">EDX</span> respectively. Possible inconsistencies
+ in the number of function arguments with the function prototype used by
+ the caller will result in fatal crash. Variable arguments lists cannot be
+ passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
+ instead.</p>
+
+ <p>GCC (GNU C Compiler) native support for this calling convention
+ is pretty fresh and it is currently present only in the recent CVS
+ versions since 21st December of 2002 which should get released as GCC
+ version 3.4. This project solved the unsupported calling convention by
+ declaration of arguments passed in registers by
+ <span class="command">__attribute__((__regparm__(3)))</span>.
+ W32 passes the arguments in registers in the order
+ <span class="constant">ECX</span>, <span class="constant">EDX</span> but
+ GCC passes them in registers <span class="constant">EAX</span>,
+ <span class="constant">EDX</span>, <span class="constant">ECX</span>.
+ This incompatibility is compensated at C source level in the
+ @{[ a_href '#functype','relaying code' ]} generated by
+ <span class="fname">captivesym</span> relay generator.</p>
+
+ @{[ doc_img 'fig/calltype_fastcall',
+ 'W32 Calling Convention <span class="constant">fastcall</span> Scheme' ]}
+
+ <table border="1" align="center">
+ <tr><td>Arguments freed by </td><td>callee</td></tr>
+ <tr><td>Arguments on the stack </td><td>#2 ... #(n-1)</td></tr>
+ <tr><td>Arguments in the registers </td><td><span class="constant">ECX</span>=#0,
+ <span class="constant">EDX</span>=#1</td></tr>
+ <tr><td>GCC ≥3.4 attribute </td><td><span class="command">__attribute__((__fastcall__))</span></td></tr>
+ <tr><td>GCC <3.4 attr. emulation</td><td><span class="command">__attribute__((__stdcall__))</span></td></tr>
+ <tr><td> </td><td><span class="command">__attribute__((__regparm__(3) /* EAX,EDX,ECX */))</span></td></tr>
+ <caption>Calling Convention <span class="constant">fastcall</span> Characteristics</caption>
+ </table>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Components page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::Components;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: Components',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<h1>Project Components</h1>
+
+ @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}
+
+ <p>Most of the work of this project is located in the single box called
+ "<span class="constant">libcaptive</span>" located in the center
+ of the scheme. This component implements the core W32 kernel API by
+ various methods described in this document.
+ The "<span class="constant">libcaptive</span>" box cannot be
+ further dissected as it is just an implementation of a set of
+ @{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym','API functions' ]}.
+ It could be separated to several subsystems such as the
+ @{[ a_href '#cache_manager','Cache Manager' ]},
+ Memory Manager, Object Manager, Runtime Library, I/O Manager
+ etc. but they have no interesting referencing structure.</p>
+
+ <p>As this project is in fact just a filesystem implementation every
+ story must begin at the device file and end at the filesystem operations
+ interface. The unified suppported interfaces are
+ <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
+ (the most low level portability, data-types and utility library for Gnome)
+ <span class="type">GIOChannel</span> (for the device access) and the custom
+ <span class="constant">libcaptive</span> filesystem API. Each of these ends
+ can be connected either to some direct interface (such as the
+ <span class="constant">captive-cmdline</span> client),
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
+ or as a general $GnomeVFS filter.
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
+ most cases as it offers standard filesystem interface by Linux kernel.
+
+ You can also use $GnomeVFS as it offers nice filter interface on
+ the UNIX user-privileges level for transparent operation with archives and
+ network protocols. This filter interface was used by this project to turn
+ the device reference such as <span class="fname">/dev/hda3</span> or <span
+ class="fname">/dev/discs/disc0/part3</span> to the fully accessible
+ filesystem (pretending being an "archive" in the device
+ reference). This device access can be specified by $GnomeVFS URLs such as:
+ <span
+ class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
+
+ <span class="constant">captive-bug-replay</span> serves just for debugging
+ purposes — you can 'replay' existing
+ <span class="fname">file.captivebug.xml.gz</span> automatically being
+ generated during W32 filesystem failure. This bugreport file will contain
+ all the touched data blocks of the device used in the moment of the
+ failure. <span class="constant">captive-bug-replay</span> will therefore
+ emulate internal virtual writable device out of these bugreported data.
+
+ <p>If the passed device reference is requested by the user to be accessed
+ either in <span class="dashdash">--ro</span> (read-only) mode or in the
+ <span class="dashdash">--rw</span> (full read-write) mode there are no
+ further device layers needed. Just in the case of <span
+ class="dashdash">--blind</span> mode another layer is involved to emulate
+ read-write device on top of the real read-only device by the method of
+ non-persistent memory buffering of all the possible write requests.</p>
+
+ <span class="constant">sandbox commit buffer</span> is involved only in the
+ case @{[ a_href '#sandbox','sandboxing feature' ]} is active. It will
+ buffer any writes to the device during the sandbox run to prevent
+ filesystem damage if the driver would fail in the meantime. If the
+ filesystem gets finally successfully unmounted this sandbox buffer can be
+ <a name="safe_flush">safely flushed</a>
+ to its underlying physical media. The buffer will be dropped
+ in the case of filesystem failure, of course. The filesystem should be
+ unmounted from time to time — it can be transparently unmounted and mounted
+ by <span class="command">commit</span> of
+ <span class="constant">captive-cmdline</span> custom client. Currently you
+ cannot force remounting when using
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
+ but it will be remounted after approx each 1MB data written automatically
+ due to @{[ a_href '#log_file_full','NTFS log file full' ]}.
+
+ Now we need to transparently
+ @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','connect' ]}
+ the device interface of <span class="type">GIOChannel</span> type through
+ @{[ a_href '#sandbox','CORBA/ORBit' ]} to the sandboxed slave.
+
+ <p>Such device is still only a UNIX style GLib <span
+ class="type">GIOChannel</span> type at this point. As we need to supply it
+ to the W32 filesystem driver we must convert it to the W32 I/O Device
+ with its capability of handling <span class="type">IRP</span>
+ (<span class="constant">I/O Request Packet</span>; structure holding the
+ request and result data for any W32 filesystem or W32 block device
+ operation)
+ requests from its upper W32 filesystem driver. Such W32 I/O Device can
+ represent either <span class="type">CD-ROM</span> or
+ <span class="type">disk</span> device type as different W32 filesystem
+ drivers require different media types — currently only
+ <span class="fname">cdfs.sys</span> requires
+ <span class="type">CD-ROM</span> type.</p>
+
+ <p>W32 media I/O Device is accessed from the W32 filesystem driver.
+ The filesystem driver itself always creates volume object by
+ <span class="function">IoCreateStreamFileObject()</span> representing the
+ underlying W32 media I/O Device as the object handled by the
+ filesystem driver itself. All the client application filesystem requests
+ must be first resolved at the filesystem structures level, passed to the
+ volume stream object of the same filesystem and then finally passed to the
+ W32 media I/O Device (already implemented by this project as an
+ interface to <span class="type">GIOChannel</span> noted above).</p>
+
+ <p>The filesystem driver is called by the core W32 kernel implementation of
+ <span class="constant">libcaptive</span> in
+ @{[ a_href '#synchronous','synchronous way' ]} in single-shot manner instead of
+ the several reentrancies while waiting for the disk I/O completions as can
+ be seen in the original
+ <span class="productname">Microsoft Windows NT</span>.
+ This single-shot synchronous behaviour is possible since all the needed
+ resources (disk blocks etc.) can be always presented as instantly ready as
+ their acquirement is solved by @{[ a_href 'hostosnote','Host-OS' ]} outside of
+ the W32 emulated @{[ a_href 'guestosnote','Guest-OS' ]} environment.
+ For several cases needed only by <span class="fname">ntfs.sys</span>
+ there had to be supported asynchronous access — parallel execution
+ is emulated by GLib <span class="function">g_idle_add_full()</span>
+ with <span class="function">g_main_context_iteration()</span> called during
+ <span class="function">KeWaitForSingleObject()</span>.</p>
+
+ <p><span class="constant">libcaptive</span> offers the W32 kernel
+ filesystem API to the upper layers. This is still not the API the common
+ W32 applications are used to as they use W32 libraries which in turn pass
+ the call to W32 kernel. For example
+ <span class="function">CreateFileA()</span> is being implemented by several
+ libraries such as <span class="fname">user32.dll</span> as a relay
+ interface for the kernel function
+ <span class="function">IoCreateFile()</span> implemented by this
+ project's <span class="constant">libcaptive</span> W32 kernel
+ emulation component.</p>
+
+ <p>As it would be very inconvenient to use the legacy, bloated and UNIX
+ style unfriendly W32 kernel filesystem API this project offers its own
+ @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
+ the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
+ This interface is supposed to be easily utilized by
+ <a href="#client_interface_customapp">a custom application accessing
+ the W32 filesystem driver</a>.</p>
+
+ <p>@{[ a_href '#sandbox','CORBA/ORBit' ]} hits us again – we need to
+ @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','translate' ]}
+ the @{[ a_href '#client_interface','custom filesystem API interface' ]}
+ out of the sandboxed slave to the UNIX space.</p>
+
+ <p><span class="constant">captive sandbox master</span> provides the
+ functionality of covering any possible sandboxed slave restarts and its
+ communication. It is also capable of
+ <a name="demultiplexing_master">demultiplexing single API operations</a>
+ to multiple its connected sandbox slaves in transparent way
+ as each of them handles
+ @{[ a_href '#mounted_one','just one filesystem device' ]}.</p>
+
+ <p>The rest of the story is not much special for this project since this is
+ a common UNIX problem how to offer user space implemented UNIX filesystem
+ as a generic system filesystem (as those are usually implemented only as
+ the components od UNIX kernel).</p>
+
+ <p>The filesystem service can be offered in several ways:</p>
+
+ <dl>
+ <dt>Custom client</dt>
+ <dd>
+ <p>One possibility would be to write
+ <a name="client_interface_customapp">a custom client application</a>
+ for this project such as file manager or a shell. Although it
+ would implement the most appropriate user interface to the set of
+ functions offered by this project (and W32 filesystem API) it has the
+ disadvantage of special client software. Appropriate client is provided
+ by this project as:
+ <span class="fname">src/client/cmdline/cmdline-captive</span></p>
+ </dd>
+
+ <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
+ <dd>
+ <p>The most usable interface is the
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
+ by <span class="constant">liblufs-captivefs</span>.
+ As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
+ already assigns separate process for each filesystem mount the
+ @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
+ is not utilized in this case.</p>
+
+ <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
+ needs multiple operating threads (each UNIX kernel operation needs
+ one free lufsd slot/thread to not to fail immediately).
+ As <span class="constant">libcaptive</span> is
+ @{[ a_href '#synchronous','single-threaded' ]} all the operations
+ get always synchronized by
+ <span class="constant">liblufs-captivefs</span>
+ before their pass over to <span class="constant">libcaptive</span>.</p>
+ </dd>
+
+ <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
+ <dd>
+ <p>This client allowing its filesystem access even without any
+ involvement of UNIX kernel from any $GnomeVFS aware client application
+ (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
+ This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
+ data flow of this project in two points — both as the lowest layer
+ device image source and also as the upper layer for the filesystem
+ operation requests.</p>
+ </dd>
+ </dl>
+
+ <p>Unimplemented and deprecated methods for providing filesystem
+ service:</p>
+
+ <dl>
+ <dt>W32 filesystem in UNIX OS kernel</dt>
+ <dd>
+ <p>The real UNIX OS filesystem implementation must be completely
+ implemented inside the hosting OS kernel. This requires special coding
+ methods with limited availability of coding features and libraries.
+ Also it would give the full system control to the untrusted W32
+ filesystem driver code with possibly fatal consequences of yet
+ unhandled W32 emulation code paths. It would benefit from the best
+ execution performance but this solution was never considered a real
+ possibility.</p>
+ </dd>
+
+ <dt>Custom NFS server</dt>
+ <dd>
+ <p>The common approach
+ <a name="offered_NFS">of filesystem implementations</a>
+ outside UNIX OS kernel were custom NFS servers usually running on the
+ same machine as the NFS-connected client as such NFS server is usually
+ an ordinary UNIX user space process. It would be possible to implement
+ this project as a custom NFS server but the NFS protocol itself
+ has a lot of fundamental flaws and complicated code for backward
+ compatibility.</p>
+ </dd>
+ </dl>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Details page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::Details;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: Implementation Details',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<h1>Implementation Details</h1>
+
+ <a name="emulmeth"><h2>Choice of the Emulation Methods</h2></a>
+
+ <p>The intent of the project was to get reliable read-write access to
+ <span class="productname">NTFS</span> partition. There are several possible
+ ways to achieve that:</p>
+
+ <a name="emulmeth_vm"><h3>Virtualmachine Running the Original W32 Subsystem</h3></a>
+
+ <p>Creating virtual-hardware PC and running the original W32 binaries
+ including their boot-loader etc. Disk device access would be passed as
+ virtual IDE disk (=hard disk drive). File access API would be implemented
+ either by special escaping by some trapped instruction out of the
+ virtualmachine while using W32 file access API or using the standard W32
+ SMB (Server Message Block) network access through some virtual network
+ card. The latter network access solution is almost the currently available
+ possibility of running full-blown disk-sharing real
+ <span class="productname">Microsoft Windows NT</span> inside virtual
+ machine emulator such as <span class="productname">VMware</span>.</p>
+
+ <p>pros: Full compatibility due to fully native codebase.</p>
+
+ <p>cons: Hard to debug, missing documentation of NT booting internals,
+ possible problems by different PC virtual-hardware than expected by NT,
+ requirement of fully installed
+ <span class="productname">Microsoft Windows NT</span> product.</p>
+
+ <a name="method_ntoskrnl"><h3>"ntoskrnl.exe" Inside Virtual Address Space</h3></a>
+
+ <p>This solution was chosen by the project. Binary filesystem driver and
+ also <span class="fname">ntoskrnl.exe</span> binary file are required.
+ Unfortunately <span class="fname">ntoskrnl.exe</span> expects a native
+ PC virtual-hardware missing during regular UNIX user space process
+ emulation, therefore such instructions must be trapped and emulated/ignored
+ from case to case.</p>
+
+ <p>Also the <a name="init_ntoskrnl">initialization code of <span
+ class="fname">ntoskrnl.exe</span></a> is not executed by this project since
+ it expects to get full PC hardware access privileges and thus some
+ datastructures do not get initialized by it (need to be trapped later at
+ runtime stage). Some of the missing initializations are solved by
+ @{[ a_href '#functype_wrap','API functions wrapping' ]}.
+
+ <p>pros: Lightweight, easier to debug.</p>
+
+ <p>cons: Possible incompatible emulation of
+ <span class="fname">ntoskrnl.exe</span> parts, missing documentation needed
+ for the implementation.</p>
+
+ <a name="emulmeth_fs"><h3>Filesystem Driver Inside Virtual Address Space</h3></a>
+
+ <p>Unlike @{[ a_href '#method_ntoskrnl','previous method' ]} here we do not use
+ even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
+ W32 is <a name="native_ntoskrnl">emulated from the project source
+ files</a>. <span class="fname">cdfs.sys</span> driver was successfuly ran
+ in this manner in the former versions of this project but the possibility
+ to run without <span class="fname">ntoskrnl.exe</span> was dropped since it
+ had no licensing gains (you need the original
+ <span class="productname">Microsoft Windows NT</span> files at least for
+ the filesystem driver itself) and the emulation of undocumented parts
+ reusable from <span class="fname">ntoskrnl.exe</span> binary was
+ a pain.</p>
+
+ <p>pros: Lightweight, easier to debug.</p>
+
+ <p>cons: Possible incompatible emulation of the whole
+ <span class="fname">ntoskrnl.exe</span>, its missing documentation.</p>
+
+
+ <a name="apichoice"><h2>API Function Implementation Choices</h2></a>
+
+ <p>During the initial point of the project development all the API
+ functions were defined as unimplemented, of course. Any call of such
+ unimplemented function is fatal and results in program termination. When we
+ need to implement any required API function we have multiple choices to do
+ so:
+ @{[ a_href '#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
+ @{[ a_href '#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
+ @{[ a_href '#functype_native_reactos','Native implementation – $ReactOS' ]},
+ @{[ a_href '#functype_native_wine','Native implementation – $Wine' ]}
+ or
+ @{[ a_href '#functype_native_libcaptive','Native implementation – project specific' ]}.
+ <!-- a_href '#functype_undef','Undefined function' -->
+
+
+ <a name="sandbox"><h2>Sandboxing of W32 filesystem</h2></a>
+
+ <p>The emulated W32 environment running the original W32 filesystem driver
+ is separated from the rest of UNIX OS. It achieves the following goals:</p>
+
+ <ul>
+ <li><b>Restartable</b>: W32 driver can be restartde in clean state if it crashed</li>
+ <li><b>Secure</b>: Malicious W32 code cannot affect the security of UNIX OS</li>
+ <li><b>Stable</b>: Buggy W32 cannot crash any part of UNIX OS</li>
+ </ul>
+
+ <p>Sandboxing is provided with the following attributes:</p>
+
+ <ul>
+ <li>standalone UNIX process with separate memory space</li>
+ <li>chroot(2) in empty directory to prevent any UNIX OS filesystem access</li>
+ <li>setuid(2) to own user/group to prevent interaction with UNIX processes</li>
+ <li>setrlimit(2) to limit system resources available for W32 environment</li>
+ <li>the only connection with the UNIX OS by CORBA/ORBit RPC</li>
+ </ul>
+
+ <p>This security is almost the same as provided by
+ emulated virtual machines such as
+ @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware' ]}.</p>
+
+ @{[ doc_img 'dia/inheritance','Sandboxing Scheme' ]}
+
+ <p>Project can be also used in non-sandboxed mode by
+ <span class="command">--no-sandbox</span> option as it is easier to debug
+ without CORBA/ORBit RPC. In this case the
+ <span class="type">DirectorySlave</span>/<span class="type">FileSlave</span>
+ options are used directly instead of their
+ <span class="type">DirectoryParent</span>/<span class="type">FileParent</span>
+ peers.</p>
+
+
+ <a name="patched"><h2>"patched" vs. "unpatched" Libraries</h2></a>
+
+ <p>Library is called <span class="constant">patched</span> if we require
+ loading its original binary code file. Project needs to patch it to be able
+ to trap all the function entry points. The only currently
+ <span class="constant">patched</span> library of this project is
+ <span class="fname">ntoskrnl.exe</span>.</p>
+
+ <p>Library is called <span class="constant">unpatched</span> if no original
+ binary code is needed since all of its functions are completely emulated by
+ @{[ a_href '#functype_native','the native implementations' ]} of this project.
+ The typical <span class="constant">unpatched</span> representative is
+ <span class="fname">hal.dll</span> as it specializes on the hardware
+ dependent code and therefore it must be completely replaced by this project
+ running in the $gnulinux operating system environment. Early versions of
+ this project had also full <span class="constant">unpatched</span>
+ <a href="#native_ntoskrnl">native implementation of
+ <span class="fname">ntoskrnl.exe</span></a> but it no longer applies.</p>
+
+ <a name="mman"><h2>Memory Management</h2></a>
+
+ <p>Original <span class="productname">Microsoft Windows NT</span>
+ architecture uses two address space areas – user space and kernel space.
+ User space is mapped in the range <span class="constant">0x00000000</span>
+ to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
+ range <span class="constant">0x80000000</span>
+ (<span class="constant">KERNEL_BASE</span> in $ReactOS sources) to
+ <span class="constant">0xFFFFFFFF</span>. All these virtual memory ranges
+ represent addresses after their MMU (Memory Management Unit) mapping, of
+ course. More discussion can be found in the
+ <a href="http://www.microsoft.com/hwdev/platform/server/PAE/PAEmem.asp">description
+ by <span class="productname">Microsoft</span></a>.</p>
+
+ <p>This project runs in the virtual address space used both for the UNIX
+ user space process part and for the W32 kernel space. Therefore this
+ project defines that W32 kernel runs in the whole range
+ <span class="constant">0x00000000</span> to
+ <span class="constant">0xFFFFFFFF</span> since there are no special mapping
+ assumptions about the UNIX user space process mapping. No W32 user space
+ exists in this project. Such approach also nullifies any special memory
+ moving operations between W32 kernel space and W32 user space memory areas
+ (such as <span class="function">MmSafeCopyToUser()</span>).</p>
+
+ <a name="unicode"><h2>Unicode Strings and Characters</h2></a>
+
+ <p>W32 platform uses 16-bit type <span class="type">wchar_t</span> while $gnulinux uses a
+ 32-bit one. This can be problem during GCC (GNU C Compiler)
+ compilation of combination of native UNIX C sources (assuming 32-bit
+ GCC with 32-bit <span class="type">wchar_t</span>) and
+ $ReactOS C sources (assuming W32 compiler with 16-bit
+ <span class="type">wchar_t</span>) for literal wide strings
+ (C source file systax: <span class="command">L"wstring"</span>).
+ Possibilities to solve this issue list:</p>
+
+ <ul>
+ <li>
+ <p>Using <span class="constant">-fshort-wchar</span> GCC option and
+ strictly differentiate between compilation of
+ <span class="productname">ReactOS</span> code and UNIX code.</p>
+
+ <p>pros: No source modifications needed, no runtime performance hit.</p>
+
+ <p>cons: No type checking if some part of code has bad compilation
+ flags, complicated way to completely split
+ <span class="productname">ReactOS</span> and UNIX code.</p>
+ </li>
+ <li>
+ <p>Wrap all <span class="productname">ReactOS</span> literal constants
+ by some conversions function call (implemented as macro
+ <span class="function">REACTOS_UCS2()</span> by this project).</p>
+
+ <p>pros: Any forgotten/mistaken conversions are type-checked and warned
+ during the compilation by GCC.</p>
+
+ <p>cons: All compiled <span class="productname">ReactOS</span> sources
+ files containing literal wide strings have to be wrapped/modified,
+ performance hit by runtime string conversions.</p>
+
+ <p>This solution was chosen to get the internal sanity checking
+ benefit.</p>
+ </li>
+ </ul>
+
+ <a name="binfmt"><h2>Supported Binary Formats</h2></a>
+
+ <p>The native W32 binary format is identified as
+ <span class="constant">PE-32</span> (Portable Executable 32-bit), such
+ files have all the usual extensions such as
+ <span class="fname">.sys</span>, <span class="fname">.exe</span>,
+ <span class="fname">.dll</span> etc. <span class="constant">PE-32</span>
+ loading support was already implemented by $ReactOS, its memory mapping
+ specifics just had to be ported to $gnulinux environment by this project.
+ This loading support does not (yet) cover importing of debug symbols from
+ W32 <span class="fname">.PDB</span> (Program DataBase) files in $gnulinux
+ ABI (Application Binary Interface) compatible way.</p>
+
+ <p>This project also supports transparent loading of UNIX
+ <span class="fname">.so</span> (Shared Object file) binary format. If you
+ have W32 source files for some W32 library you can try to compile it by GCC
+ to get the shared library with $gnulinux ABI compatible debug information
+ (GCC option <span class="constant">-ggdb3</span> recommended). Beware of
+ possible compilation problems as <span class="productname">Microsoft</span>
+ C code expects <span class="constant">exception</span> handling to be
+ supported by the compiler (definitely not the case of the plain C compiler
+ of GCC) — all the exception catching code should be discarded as any
+ @{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
+ such driver is running in the scope of this project. You can use the
+ following script of this project to compile W32 filesystem source files as
+ UNIX <span class="fname">.so</span>:
+ @{[ captive_srcfile 'src/w32-mod/ext2fsd.so-build.sh' ]}</p>
+
+ <p>Be aware of some differences if you use
+ <span class="constant">PE-32</span> binary format file vs.
+ <span class="fname">.so</span> format file.
+ <span class="constant">PE-32</span> use the appropriate W32 specific
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
+ <span class="fname">.so</span> must be completely compiled in the standard
+ UNIX @{[ a_href '#calltype_cdecl','cdecl call type semantics' ]}.
+ @{[ a_href '#functype_native','Native function implementations' ]} do not need
+ to be explicitely exported by <span class="fname">captivesym</span> as they
+ are resolved automatically by the UNIX dynamic system linker. It may be
+ surprising you will have to fix all such missing symbol exports if you
+ advance during the development from the debugging
+ <span class="fname">.so</span> file for the production version of the
+ original <span class="constant">PE-32</span> binary file.</p>
+
+
+ <a name="mounted_one"><h2>At Most One Mounted Filesystem</h2></a>
+
+ <p>The project technically supports only one (exactly one...) mounted
+ filesystem device and only one filesystem driver. There is nothing
+ complicated to support multiple disks and multiple loaded filesystem
+ modules but as they would share the address space it would only bring
+ a possible complications during bug reports and the bug solving
+ itself. It was considered as a more sane way to support multiple W32
+ mounted disks by completely separately running project instances in
+ a different UNIX processes communicating from their sandboxes via
+ @{[ a_href '#sandbox','CORBA sandbox interface' ]}. This sandboxing
+ feature is not yet deployed although its code is already prepared.</p>
+
+ <p>The project also does not support any state cleanup to be able to load
+ filesystem <span class="constant">A</span>,
+ cleanup <span class="constant">A</span> and load a different
+ filesystem <span class="constant">B</span> in the same process address
+ space. It complies with the preventions of the possible debugging
+ complications as noted above. Despite this you still must call the function
+ <span class="function">captive_shutdown()</span> to flush all the pending
+ filesystem buffers to the disk. After calling
+ <span class="function">captive_shutdown()</span> the process address space is
+ no longer usable for any further project operations and the process is
+ expected to be terminated in the manner compatible with its driving
+ @{[ a_href '#sandbox','CORBA sandbox interface' ]} control master.</p>
+
+ <p>Each sandbox executing the untrusted W32 binary filesystem driver code
+ is connected through its
+ @{[ a_href '#sandbox','CORBA sandbox interface' ]} at the point of upper
+ layer <span class="constant">libcaptive</span>-specific filesystem API, at
+ the point of the bottom layer of <span class="type">GIOChannel</span>
+ device access and also for transfers of GLib logging
+ messages/warnings/errors out of the sandbox to the user.</p>
+
+
+ <a name="synchronous"><h2>Multithreading and Multiple Processors</h2></a>
+
+ <p>W32 platform stands on its thorough architecture parallelism. It
+ must lock all its objects to maintain coherence in presence of
+ multithreading and multiple processors. Since the author of this project
+ considers any parallel execution a serious obstacle for debugging the whole
+ project architecture was designed to prevent any undeterministic behaviour.
+ Therefore this projects always emulates uniprocessor
+ <span class="productname">Microsoft Windows NT</span> kernel
+ (<span class="constant">KeNumberProcessors</span> symbol is always 1),
+ everything runs in the single initial thread/process and all the filesystem
+ operations are performed as synchronous
+ ("synchronous" by flags
+ <span class="constant">FILE_SYNCHRONOUS_IO_ALERT</span>,
+ <span class="constant">FO_SYNCHRONOUS_IO</span>,
+ <span class="constant">IRP_SYNCHRONOUS_API</span>,
+ <span class="constant">IRP_SYNCHRONOUS_PAGING_IO</span>,
+ forced <span class="constant">TRUE</span> result of
+ <span class="function">IoIsOperationSynchronous()</span>
+ etc.).
+ For several cases needed only by <span class="fname">ntfs.sys</span> there
+ had to be supported asynchronous access
+ (<span class="constant">STATUS_PENDING</span> return code) – parallel
+ execution is emulated by GLib
+ <span class="function">g_idle_add_full()</span> with
+ <span class="function">g_main_context_iteration()</span> called during
+ <span class="function">KeWaitForSingleObject()</span>.</p>
+ Since there is a possibility a real W32 parallel threading would
+ be yet needed in the future all the code that would be hit by W32
+ multithreading capability is marked by
+ <span class="constant">TODO:thread</span> comment.</p>
+
+ <p>Multiple processors (SMP) support will never need to be implemented
+ since uniprocessor W32 kernels apparently run the filesystem driver modules
+ fine. As this project implements only the uniprocessor W32 kernel all the
+ processor locking functions and structures such as
+ <span class="constant">KSPIN_LOCK</span> etc. can be safely implemented as
+ no-operations.</p>
+
+ <p>Asynchronous callbacks registered for
+ <span class="constant">IO_WORKITEM</span>s are passed as GLib idle
+ functions by <span class="function">g_idle_add_full()</span>. Although they
+ will probably never be executed during non-interactive project's batch
+ executions it is the responsibility of W32 driver implementation to
+ complete all the pending tasks before its W32 shutdown. Such W32 shutdown
+ is done during cleanup of the project's execution by
+ <span class="function">captive_shutdown()</span>.</p>
+
+ <a name="paranoia"><h2>Paranoia Checks</h2></a>
+
+ <p>A general approach of software projects development is to implement
+ many internal sanity checks during the development stage but to produce the
+ most optimized final release product without those debugging checks.</p>
+
+ <p>Facilities for these practices can be seen in the standard
+ C include files for example as function
+ <span class="function">assert()</span> which gets disabled by the
+ <span class="constant">NDEBUG</span> symbol used during the final optimized
+ executable compilation. This project uses Gnome GLib messaging subsystem
+ offering sanity checks discarded by symbols
+ <span class="constant">G_DISABLE_ASSERT</span> and
+ <span class="constant">G_DISABLE_CHECKS</span>.
+ <span class="productname">Microsoft</span> also produces two versions of
+ its products – regular customers use the "free build" (also
+ called "retail") while the programmers should develop their code
+ on the "checked build" product releases.</p>
+
+ <p>As this project will always run unknown binary code of proprietary W32
+ filesystem drivers, the code can never be trusted. Such code even runs in
+ the same unprotected address space as its controlling UNIX code. Since
+ there is not enough documentation for the W32 components of the system and
+ also such documentation is usually misleading it can never be considered as
+ 100% emulation. Even in the final releases all the sanity checks
+ implemented in this project should remain active as all the project's code
+ always interacts with unknown and untrusted W32 binaries.</p>
+
+ <p><span class="productname">Microsoft Windows NT</span> code is written in
+ a foolproof style as it accepts even invalid input values, and which
+ it usually corrects. This makes long-term debugging a pain as it hides
+ sources of problems. "Checked build" releases were probably
+ designed to fix this flaw by strict consistency checks but it did not reach
+ its goals as such checks are usually missing in the code.</p>
+
+ <p>This project has strict consistency checks across all the code to make
+ the debugging phase easy enough. Failed sanity check is not always
+ a bug – sometimes it just means the real W32 binary code is more
+ benevolent than it could be expected according to the documentation and
+ such sanity check gets removed for the next version build. In other cases
+ the failed sanity checks mean the execution path for some unexpected
+ arguments combination was not yet implemented by this project. I may also
+ mean a bug, of course...</p>
+
+ <p>Last but not least – never miss a possible sanity check as its
+ later removal is in an order of magnitude cheaper than an uncaught
+ invalid assumption. Failed assertion is not always a bug although it
+ has to be fixed, of course.</p>
+
+
+ <a name="logfile"><h2>STATUS_LOG_FILE_FULL</h2></a>
+
+ <p>After writing approx. 1MB of data on NTFS test partition NTFS driver
+ returns for any further write requests
+ <span class="constant">STATUS_LOG_FILE_FULL</status> error code.
+ Apparently it is caused by the fact this project is
+ @{[ a_href '#synchronous','single-threaded' ]} and it ignores the spawn
+ of parallel journalling thread during <span class="fname">ntfs.sys</span>
+ initialization.</p>
+
+ <p>Fortunately <span class="fname">ntfs.sys</span> will clear its
+ journalling log file during filesystem unmount. This project will therefore
+ remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</status>
+ is detected to workaround missing journalling thread.</p>
+
+ <p>Similiar behaviour can be seen during write of compressed files —
+ the file gets written uncompressed and its compression will proceed only
+ during the final filesystem unmount.</p>
+
+ <p>For these reasons it was mandatory to support
+ @{[ a_href '#parent_connector','transparent volume remounting' ]}.</p>
+
+
+ <a name="parent_connector"><h2><span class="constant">ParentConnector</span> volume remounter</h2></a>
+
+ <p>The sandbox master component of this project has control of restarting
+ its sandbox slaves containing the W32 filesystem. Target goal of
+ <span class="constant">ParentConnector</span> component is to transparently
+ provide persistent view of files and directories over the sandboxed slaves
+ being restarted.</p>
+
+ <p>In the case of read-only operations it would be simple as we could only
+ save our state of currently opened filesystem objects with their read
+ file/directory offset. Write operations can be handled as the read-only
+ ones as long as all the operations are successful. In the case of W32
+ filesystem crash we loose all the past write operations. If we would redo
+ all the write operations we could very easily invoke the same crash.
+ Therefore we write:</p>
+
+ <blockquote class="command">
+ <p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
+ </blockquote>
+
+ <p>message to syslog and refuse any further operations with this
+ object.</p>
+
+ @{[ doc_img 'dia/parent-connector','Parent Connector' ]}
+
+ <p><span class="constant">HANDLE</span> represents W32 object open in
+ existing W32 filesystem.<span class="constant">HANDLE</span> is created
+ on-demand according to the saved state of the object (such as its
+ pathname). Even the whole <span class="constant">VFS</span> sandbox slave
+ is spawn on-demand if some object operation requests it.</p>
+
+ <p>W32 filesystem crash can obviously occur at any moment - it generates
+ @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
+ @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
+ <span class="constant">abort</span>. Successful filesystem unmount
+ (even as the part of remount operation) must be first preceded by
+ <span class="constant">detach</span> signal to close all existing
+ W32 <span class="constant">HANDLE</span>s. After their close the filesystem
+ gets the unmount requests. Only in the case all the close operations
+ succeeded including the final filesystem unmount the signal
+ <span class="constant">cease</span> can be activated to notify all the
+ dirty (written) objects they are now clean. During this
+ <span class="constant">cease</span> signal the project will also
+ @{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
+ underlying media.</p>
+
+ <p>Objects never written remain in <span class="constant">clean</span>
+ state and they can be transparently reopened even if W32 filesystem crash
+ occurs.</p>
+
+
+HERE
+
+
+My::Web->footer();
BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
use My::Web;
require "CGI";
+use project::captive::doc::Macros;
My::Web->init(
"__PACKAGE__"=>__PACKAGE__,
- "title"=>'Captive NTFS doc',
- "head_css"=>"
-.productname { font-family: cursive; }
-.fname { font-family: monospace; }
-.constant { font-family: monospace; }
-.author { font-family: cursive; }
-.stuff { font-style: italic; font-size: larger; margin-left: 20%; margin-right: 10%; }
-.function { font-family: monospace; }
-.type { font-family: monospace; }
-.command { font-family: monospace; }
-.instruction { font-style: italic; }
-",
+ "title"=>'Captive NTFS Developer Documentation',
+ "head_css"=>$doc_Macros_head_css,
);
My::Web->heading();
-sub doc_img ($$)
-{
-my($img_base,$caption)=@_;
-
- my $r="";
- $r.='<table border="0" align="center">'."\n";
- $r.="\t<tr><td>".img($img_base,$caption)."</td></tr>\n";
- $r.="\t<caption>$caption</caption>\n";
- $r.='</table>'."\n";
- $r.=vskip "2ex";
- return $r;
-}
-
-sub captive_srcfile ($;$)
-{
-my($filename,$text)=@_;
-
- a_href 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/'.$filename.'?rev=HEAD',
- ($text || $filename);
-}
-
-my $freespeech=a_href 'http://www.gnu.org/philosophy/free-sw.html','Free';
-my $freebeer=a_href 'http://www.gnu.org/philosophy/free-sw.html','free (as in beer)';
-
-sub productname
-{
-my($url,$name)=@_;
-
- return '<span class="productname">'.a_href($url,CGI::escapeHTML($name)).'</span>';
-}
-my $Wine=productname 'http://www.winehq.com/','Wine';
-my $ReactOS=productname 'http://www.reactos.com/','ReactOS';
-my $LinuxNTFS=productname 'http://linux-ntfs.sourceforge.net/','Linux NTFS';
-my $GnomeVFS=productname 'http://developer.gnome.org/doc/API/gnome-vfs/','Gnome-VFS';
-my $GnomeVFSmodule=productname 'http://developer.gnome.org/doc/API/gnome-vfs/modules.html','Gnome-VFS-module';
-my $gnulinux='GNU/Linux';
-
-
print <<"HERE";
-<h1>About</h1>
-
- <h2>Reasons for the Implementation</h2>
-
- <p>Currently there is no possibility to any of the available $freespeech
- ($freespeech used in the following text in the meaning of
- "@{[ a_href 'http://www.gnu.org/philosophy/free-sw.html','free as in speech' ]}")
- operating systems to reliably write to the most common disk partition
- filesystem type – <span class="productname">Microsoft NTFS</span>. It would
- be already supported a long time ago but there is no proper documentation of
- <span class="productname">NTFS</span> filesystem data structures available.
- Since <span class="productname">Microsoft</span> corporation continues in its
- propagation of <span class="productname">Microsoft Windows NT</span>
- (<span class="productname">NT</span> identifier used in the following text
- applies to all the products of <span class="productname">Microsoft</span>
- <span class="productname">NT</span> series such as
- <span class="productname">NT 4.0</span>,
- <span class="productname">2000</span> as NT-5.0
- and
- <span class="productname">XP</span> as NT-5.1.)
- based operating systems <span class="productname">NTFS</span> is the default
- disk file system type for vendor preinstalled <span class="productname">Microsoft Windows</span>.
-
- <p>Unfortunately the <span class="productname">NTFS</span> filesystem has too
- complex data structure to allow a complete reverse enginnering process in
- reasonable time. Currently available $freespeech solutions such as $LinuxNTFS
- filesystem have already implemented reliable reverse
- engineered read-only access. However <a name="reliability">reliabile</a>
- read-write part of the access would require much better
- knowledge of the <span class="productname">NTFS</span> data structures.
- Currently only rewriting of already existing file data blocks is supported
- by $LinuxNTFS — no file creation, no file deletion, no directory operations etc.
- Also any future versions of <span class="productname">NTFS</span> filesystem
- would require another major reverse engineering effort.</p>
+<ul>
+<li><a href="About.html.pl">About</a>
+ <ul>
+ <li><a href="About.html.pl#reasons">Reasons for the Implementation</a></li>
+ <li><a href="About.html.pl#challenges">Challenges of the Project</a></li>
+ <li><a href="About.html.pl#versions">Microsoft Windows Versions Compatibility</a></li>
+ </ul></li>
- <h2>Challenges of the Project</h2>
-
- <p>The <a name="NTFSgoal">ultimate goal</a> of this project is definitely the
- free implementation of @{[ a_href '#reliability','reliable' ]} read-write <span
- class="productname">NTFS</span> filesystem driver. This project chose to
- solve this problem in the style of $Wine project by using the original binary
- <span class="fname">ntfs.sys</span> and emulating all the required layers of
- <span class="productname">Microsoft Windows NT</span> for it.</p>
-
- <p>Unfortunately this effort is tainted by only partial and generally
- insufficient documentation of API between filesystem driver
- (<span class="fname">ntfs.sys</span>) and the
- <span class="productname">Microsoft Windows NT</span>
- ("@{[ a_href 'http://mail.gnu.org/archive/html/libtool/2000-09/msg00000.html','W32' ]}"
- in the following text) kernel <span class="fname">ntoskrnl.exe</span>. Note
- that this API is a different than the one being used in the $Wine project
- since <span class="productname">Wine</span> implements only the user space
- part of W32.</p>
-
-
- <h2>Microsoft Windows Versions Compatibility</h2>
-
- <p>Currently this project supports only driver files of
- <span class="productname">Microsoft Windows XP</span> (NT-5.1)
- in the following releases:</p>
-
+<li><a href="Architecture.html.pl">Architecture</a>
+ <ul>
+ <li><a href="Architecture.html.pl#existing_emulation">Existing Emulation Projects</a></li>
+ <li><a href="Architecture.html.pl#law">Laws and Licensing Conditions</a>
<ul>
- <li>Microsoft Windows XP Service Pack 1a Checked Build U.S.</li>
- <li>Microsoft Windows XP Service Pack 1a Free Build U.S.</li>
- <li>Microsoft Windows XP (No Service Pack) Checked Build U.S.</li>
- <li>Microsoft Windows XP (No Service Pack) Free Build U.S.</li>
- </ul>
-
- <p>The latest list of supported driver files can be found in:
- @{[ captive_srcfile 'src/install/acquire/w32-mod-id.captivemodid.xml' ]}</p>
-
- <p>Microsoft Windows NTFS filesystem driver is capable of accessing even
- the older formats of the filesystem. Porting to Microsoft Windows 2003
- Server is expected to be done soon. There can be also a danger of Microsoft
- Windows upgrading NTFS disk filesystem where you would no longer be able
- to access your NTFS disk by your original
- <span class="productname">Microsoft Windows</span> version.
- This upgrade does not happen as it occurs only during complete CD-ROM
- Microsoft Windows system installation – such operation is not performed
- by this project.</p>
-
- <p>There is no problem technically obtaining the needed driver files of
- <span class="productname">Microsoft Windows XP</span> as they are freely
- downloadable at:
- @{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp' ]}</p>
- There may be @{[ a_href '#law','legal reasons' ]} you would not be allowed
- to use there files if you own license to a different version of
- <span class="productname">Microsoft Windows</span>. Legal rights will very
- depending on your country.</p>
-
-
-<h1>Architecture</h1>
-
- <p>The principle of the
- project lies in the glue between
- <span class="productname">Microsoft Windows NT</span> kernel space
- environment and $gnulinux user space process environment:</p>
-
- @{[ doc_img 'arch-W32','Microsoft Windows Subsystems Architecture' ]}
- @{[ doc_img 'arch-captive','Captive Subsystems Architecture' ]}
-
- <a name="existing_emulation"><h2>Existing Emulation Projects</h2></a>
-
- <p>There were two well-known $freespeech projects emulating W32 subsystems
- to reach the compatibility with various W32 components:
- $Wine and $ReactOS. Sad moment is that the goals of this project do not fit
- very well into any role in those two ones. Therefore this project went
- its own way of emulation:</p>
-
- <table align="center" border="1">
- <tr>
- <th>@{[ a_href '#guestosnote','Guest-OS' ]}</th>
- <th>@{[ a_href '#hostosnote' ,'Host-OS' ]}</th>
- <th>Implements</th>
- <th>W32 kernel library</th>
- </tr>
- <tr>
- <td>$Wine</td>
- <td>$gnulinux</td>
- <td>W32 user space</td>
- <td><span class="fname">ntdll.dll</span></td>
- </tr>
- <tr>
- <td>$ReactOS</td>
- <td><span class="constant">i386</span> hardware</td>
- <td>W32 kernel and user space</td>
- <td><span class="fname">ntoskrnl.exe</span></td>
- </tr>
- <tr style="height: 1ex;"></tr>
- <tr>
- <td>this project</td>
- <td>$gnulinux</td>
- <td>W32 kernel</td>
- <td><span class="fname">ntoskrnl.exe</span></td>
- </tr>
- <caption>Emulation Projects Characteristics</caption>
- </table>
-
- <dl>
- <a name="guestosnote"><dt>Guest-OS</dt></a>
- <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#guestos','Guest OS' ]}:
- An operating system that runs inside a virtual machine.</dd>
- <a name="hostosnote" ><dt>Host OS</dt></a>
- <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#hostos' ,'Host OS' ]}:
- An operating system that runs on the host machine.</dd>
- </dl>
-
- <p>While $ReactOS provides the necessary W32 kernel subsystem emulation
- code we also need to run such @{[ a_href '#guestosnote','Guest-OS' ]} in the
- @{[ a_href '#hostosnote','Host-OS' ]} $gnulinux. Initially it was planned to
- extend $Wine with the W32 kernel space emulation functionality but
- fortunately <span class="author">Steven Edwards</span> pointed to the $ReactOS
- which better suits the needs of this project by its already implemented W32
- kernel space emulation.</p>
-
- <p>The <a name="reactos_nocare">original reasons</a> for developing
- $ReactOS still make no sense to the author of this project. Free
- implementation of W32 platform standalone running on the machine hardware
- is no longer free as most od the W32 applications are usually closed source
- and the user still looses its freedom on the application level anyway. Even
- in the case of available free applications there still remains the
- disadvantage of loosing the Host-OS platform availability if implemented in
- the $Wine style. For these ideology incompatibilities not much effort was
- made for acceptance the fixes and improvements of $ReactOS by this project.
- Moreover new functionality is not being implemented to the $ReactOS part
- but it is coded in Gnome style in the project specific source files
- place.</p>
-
- <p>The most serious problem of $ReactOS is its dependence on the direct
- <span class="constant">i386</span> hardware instead of some
- @{[ a_href '#hostosnote','Host-OS' ]} as required by the goals of this project.
- W32 is designed to be hardware-independent using its
- <span class="fname">hal.dll</span>. Unfortunately $ReactOS does not follow
- this design and thus there are needed various patches and replaces of its
- various parts and its hardware-dependent code. Despite it $ReactOS code
- base still made a big asset for this project.</p>
-
- <p class="stuff">... and @{[ a_href 'http://www.reactos.com/','ReactOS' ]} cannot run on Linux!<br />
-
-
-
- <p>Some API functions are provided both by
- <span class="fname">ntdll.dll</span> and
- <span class="fname">ntoskrnl.exe</span> in W32.
- <span class="author">Casper Hornstrup</span> enlightened such functions
- calling conventions have to be differentiated as
- <span class="fname">ntdll.dll</span> lives in the user space (low address
- space – below <span class="constant">0x80000000</span>) and
- <span class="fname">ntoskrnl.exe</span> in the kernel space (high address
- space – above <span class="constant">0x80000000</span>). Although they
- contain slightly different set of symbols (functions)
- <span class="fname">ntdll.dll</span> still can be considered as a user
- space interface to the kernel space implementation by
- <span class="fname">ntoskrnl.exe</span>.</p>
-
- <p>Currently there are
- no plans to ever extend the project's crossplatformity beyond the
- <span class="constant">i386</span> processor
- (<span class="constant">i386</span> used here as
- @{[ a_href 'http://www.intel.com/','Intel' ]} architecture covering 32-bit
- processors compatible with <span class="constant">i386</span>,
- <span class="constant">i486</span>, ...).</p>
-
- <h2>API Function Implementation Choices</h2>
-
- <p>During the initial point of the project development all the API
- functions were defined as unimplemented, of course. Any call of such
- unimplemented function is fatal and results in program termination. When we
- need to implement any required API function we have multiple choices to do
- so:
- @{[ a_href '#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
- @{[ a_href '#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
- @{[ a_href '#functype_native_reactos','Native implementation – $ReactOS' ]},
- @{[ a_href '#functype_native_wine','Native implementation – $Wine' ]}
- or
- @{[ a_href '#functype_native_libcaptive','Native implementation – project specific' ]}.
- <!-- a_href '#functype_undef','Undefined function' -->
-
- <h2>"patched" vs. "unpatched" Libraries</h2>
-
- <p>Library is called <span class="constant">patched</span> if we require
- loading its original binary code file. Project needs to patch it to be able
- to trap all the function entry points. The only currently
- <span class="constant">patched</span> library of this project is
- <span class="fname">ntoskrnl.exe</span>.</p>
-
- <p>Library is called <span class="constant">unpatched</span> if no original
- binary code is needed since all of its functions are completely emulated by
- @{[ a_href '#functype_native','the native implementations' ]} of this project.
- The typical <span class="constant">unpatched</span> representative is
- <span class="fname">hal.dll</span> as it specializes on the hardware
- dependent code and therefore it must be completely replaced by this project
- running in the $gnulinux operating system environment. Early versions of
- this project had also full <span class="constant">unpatched</span>
- <a href="#native_ntoskrnl">native implementation of
- <span class="fname">ntoskrnl.exe</span></a> but it no longer applies.</p>
-
- <h2>Memory Management</h2>
-
- <p>Original <span class="productname">Microsoft Windows NT</span>
- architecture uses two address space areas – user space and kernel space.
- User space is mapped in the range <span class="constant">0x00000000</span>
- to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
- range <span class="constant">0x80000000</span>
- (<span class="constant">KERNEL_BASE</span> in $ReactOS sources) to
- <span class="constant">0xFFFFFFFF</span>. All these virtual memory ranges
- represent addresses after their MMU (Memory Management Unit) mapping, of
- course. More discussion can be found in the
- <a href="http://www.microsoft.com/hwdev/platform/server/PAE/PAEmem.asp">description
- by <span class="productname">Microsoft</span></a>.</p>
+ <li><a href="Architecture.html.pl#law_servicepack">Microsoft Service Pack</a></li>
+ </ul></li>
- <p>This project runs in the virtual address space used both for the UNIX
- user space process part and for the W32 kernel space. Therefore this
- project defines that W32 kernel runs in the whole range
- <span class="constant">0x00000000</span> to
- <span class="constant">0xFFFFFFFF</span> since there are no special mapping
- assumptions about the UNIX user space process mapping. No W32 user space
- exists in this project. Such approach also nullifies any special memory
- moving operations between W32 kernel space and W32 user space memory areas
- (such as <span class="function">MmSafeCopyToUser()</span>).</p>
-
- <h2>Unicode Strings and Characters</h2>
-
- <p>W32 platform uses 16-bit type <span class="type">wchar_t</span> while $gnulinux uses a
- 32-bit one. This can be problem during GCC (GNU C Compiler)
- compilation of combination of native UNIX C sources (assuming 32-bit
- GCC with 32-bit <span class="type">wchar_t</span>) and
- $ReactOS C sources (assuming W32 compiler with 16-bit
- <span class="type">wchar_t</span>) for literal wide strings
- (C source file systax: <span class="command">L"wstring"</span>).
- Possibilities to solve this issue list:</p>
+ <li><a href="Components.html.pl">Project Components</a></li>
+ <li><a href="Reverse.html.pl">Reverse Engineering</a>
<ul>
- <li>
- <p>Using <span class="constant">-fshort-wchar</span> GCC option and
- strictly differentiate between compilation of
- <span class="productname">ReactOS</span> code and UNIX code.</p>
-
- <p>pros: No source modifications needed, no runtime performance hit.</p>
-
- <p>cons: No type checking if some part of code has bad compilation
- flags, complicated way to completely split
- <span class="productname">ReactOS</span> and UNIX code.</p>
- </li>
- <li>
- <p>Wrap all <span class="productname">ReactOS</span> literal constants
- by some conversions function call (implemented as macro
- <span class="function">REACTOS_UCS2()</span> by this project).</p>
-
- <p>pros: Any forgotten/mistaken conversions are type-checked and warned
- during the compilation by GCC.</p>
-
- <p>cons: All compiled <span class="productname">ReactOS</span> sources
- files containing literal wide strings have to be wrapped/modified,
- performance hit by runtime string conversions.</p>
-
- <p>This solution was chosen to get the internal sanity checking
- benefit.</p>
- </li>
- </ul>
-
- <h2>Supported Binary Formats</h2>
-
- <p>The native W32 binary format is identified as
- <span class="constant">PE-32</span> (Portable Executable 32-bit), such
- files have all the usual extensions such as
- <span class="fname">.sys</span>, <span class="fname">.exe</span>,
- <span class="fname">.dll</span> etc. <span class="constant">PE-32</span>
- loading support was already implemented by $ReactOS, its memory mapping
- specifics just had to be ported to $gnulinux environment by this project.
- This loading support does not (yet) cover importing of debug symbols from
- W32 <span class="fname">.PDB</span> (Program DataBase) files in $gnulinux
- ABI (Application Binary Interface) compatible way.</p>
-
- <p>This project also supports transparent loading of UNIX
- <span class="fname">.so</span> (Shared Object file) binary format. If you
- have W32 source files for some W32 library you can try to compile it by GCC
- to get the shared library with $gnulinux ABI compatible debug information
- (GCC option <span class="constant">-ggdb3</span> recommended). Beware of
- possible compilation problems as <span class="productname">Microsoft</span>
- C code expects <span class="constant">exception</span> handling to be
- supported by the compiler (definitely not the case of the plain C compiler
- of GCC) — all the exception catching code should be discarded as any
- @{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
- such driver is running in the scope of this project. You can use the
- following script of this project to compile W32 filesystem source files as
- UNIX <span class="fname">.so</span>:
- @{[ captive_srcfile 'src/w32-mod/ext2fsd.so-build.sh' ]}</p>
-
- <p>Be aware of some differences if you use
- <span class="constant">PE-32</span> binary format file vs.
- <span class="fname">.so</span> format file.
- <span class="constant">PE-32</span> use the appropriate W32 specific
- @{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
- <span class="fname">.so</span> must be completely compiled in the standard
- UNIX @{[ a_href '#calltype_cdecl','cdecl call type semantics' ]}.
- @{[ a_href '#functype_native','Native function implementations' ]} do not need
- to be explicitely exported by <span class="fname">captivesym</span> as they
- are resolved automatically by the UNIX dynamic system linker. It may be
- surprising you will have to fix all such missing symbol exports if you
- advance during the development from the debugging
- <span class="fname">.so</span> file for the production version of the
- original <span class="constant">PE-32</span> binary file.</p>
-
- <a name="reverse"><h2>Reverse Engineering</h2></a>
-
- <p>This project has no intentions to reverse engineer and document the
- filesystem data structures themselves since they are being encapsulated by
- the filesystem driver. For these reasons the resources available in
- projects such as $LinuxNTFS get out of any possible use. This project goal
- is to provide fully compatible API interface to the rest of the W32 system
- to persuade the filesystem driver it is running in the native
- <span class="productname">Microsoft Windows XP</span> environment.</p>
-
- <p>All the W32 filesystem drivers are running in the W32 kernel address
- space and this area of W32 API is not much documented by
- <span class="productname">Microsoft</span>. Some API functions are not
- documented at all and the others are documented insufficiently for a their
- possibly needed reimplementation from scratch. Documentation being
- consulted primarily consists of
- <span class="productname">@{[ a_href 'http://msdn.microsoft.com/library/default.asp?url=/library/en-us/kmarch/hh/kmarch/kmhdr_6enb.asp','MSDN (Microsoft Developer Network) Kernel-Mode Driver Architecture: Windows DDK' ]}</span>
- documentation and also various other 3rd party documentation resources such as
- <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
- 'The NT Cache Manager Description' ]}</span>,
- <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
- 'Learn About NT'."'".'s File-system Cache' ]}</span>,
- <span class="productname">@{[ a_href 'http://www.ntfsd.org/archive/',
- 'NT File System Developers mailing list archives' ]}</span>
- including various
- @{[ a_href 'http://www.google.com/search?q=site%3Amicrosoft.com','fulltext searches' ]}
- through Internet from case to case.</p>
-
- <p>Sometimes no sufficient documentation was found and some code behaviour
- had to be reverse engineered directly from the binaries of
- <span class="fname">ntoskrnl.exe</span>,
- <span class="fname">cdfs.sys</span>,
- <span class="fname">fastfat.sys</span>
- and primarily
- <span class="fname">ntfs.sys</span>.
- Up to now the code was disassembled by
- <span class="productname">@{[ a_href 'http://www.simtel.net/pub/pd/29498.html','IDA Freeware' ]}</span>
- and by
- <span class="productname">dumpbin.exe</span> of
- <span class="productname">Microsoft Visual Studio</span>.
- <span class="productname">dumpbin.exe</span> is fortunately able to
- interpret debug symbols from W32 <span class="fname">.PDB</span>
- (Program DataBase) debug information files.</p>
-
- <h3><span class="productname">dumpbin.exe</span>:</p></h3>
-
- <p>You should use the following options for
- <span class="productname">dumpbin.exe</span>:</p>
-
- <blockquote class="command">
- <p>dumpbin.exe /all /rawdata:none /disasm /pdbpath:verbose FILENAME.SYS</p>
- </blockquote>
-
- <p>You should see the following line in the output:</p>
-
- <blockquote class="command">
- <p>PDB file found at '.\\FILENAME.pdb'</p>
- </blockquote>
-
- <a name="WinDbg"><h3><span class="productname">WinDbg</span> Windows NT kernel debugging</h3></a>
-
- <p><span class="productname">WinDbg</span> is downloadable from:
- @{[ a_href 'http://www.microsoft.com/whdc/ddk/debugging/installx86.mspx' ]}</p>
-
- <p>This is (the only?) tool able to debug filesystem drivers incl.
- <span class="fname">ntfs.sys</span>. You will need two computers running
- <span class="productname">Microsoft Windows</span> — one computer will run
- <span class="productname">WinDbg</span> while the other one will be
- frozen in remote Windows NT kernel debug mode. It does not matter which
- <span class="productname">Microsoft Windows</span> version will be run
- on the <span class="productname">WinDbg</span> side.</p>
-
- <p>The most easy way to setup two computers is to use commercial
- <span class="productname">@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</span>
- where you can run two virtual machines simultaneously on single PC
- hardware and you can connect them by a virtual serial port provided by
- <span class="productname">VMware</span>.</p>
-
- <h4><span class="productname">WinDbg</span> side setup</h4>
-
- @{[ doc_img 'ntdebug-vmware-windbg',
- '<span class="productname">VMware</span> virtual serial port'
- .' of <span class="productname">WinDbg</span> side' ]}
-
- <p>You should setup <span class="productname">WinDbg</span> according
- to:</p>
-
- @{[ doc_img 'ntdebug-windbg-port','Port settings of <span class="productname">WinDbg</span>' ]}
- @{[ doc_img 'ntdebug-windbg-sym','Symbols files location of <span class="productname">WinDbg</span>' ]}
-
- <span class="constant">Symbols</span> should point to the directory where
- reside files extracted from the symbol archive for your version of
- <span class="productname">Microsoft Windows</span>. In the case of the
- recommended <span class="productname">Microsoft Windows XP Service Pack 1 Checked Build</span>
- you should use:
- @{[ a_href 'http://msdl.microsoft.com/download/symbols/packages/windowsxp/xpsp1sym_x86_chk.exe' ]}</p>
-
- <blockquote class="command">
- <p># Rename xpsp1sym_x86_chk.exe contents .pdb files for WinDbg<br />
- @{[ CGI::escapeHTML(q{for i in *.pdb*;do ext="`echo $i|sed 's/^.*\.pdb\.\(.*\)$/\1/'`";if [ "$i" = "$ext" ];then echo "BAD:$i";break;fi;base="`echo $i|sed 's/\(\.pdb\)\..*$/\1/'`";echo "md $ext";echo "move /-y $i $ext\\$base";done|sort -u|sed 's/$/'`echo -ne '\r'`'/g' >/tmp/rename.bat}) ]}</p>
- </blockquote>
-
- <p>The resulting <span class="command">rename.bat</span> for
- <span class="command">xpsp1sym_x86_chk.exe</span> can be found at:
- @{[ a_href 'xpsp1sym_x86_chk-rename.bat.zip' ]}</p>
-
- <p>The resulting directory should contain at least
- <span class="command">sys\\ntfs.pdb</span>
- and
- <span class="command">exe\\ntoskrnl.pdb</span>.</p>
-
- <p>Your successfuly connected target (after the steps described
- below) should look like:</p>
-
- @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}
-
- <h4>Setup of the side being kernel-debugged</h4>
-
- @{[ doc_img 'ntdebug-vmware-xpdebug',
- '<span class="productname">VMware</span> virtual serial port'
- .' of the side being kernel-debugged' ]}
-
- <p>You must use the following options in your
- <span class="command">c:\\boot.init</span> command-line:</p>
-
- <blockquote class="command">
- <p>/debug /debugport=COM1 /baudrate=115200</p>
- </blockquote>
-
- <p>After booting this <span class="command">boot.ini</span>-entry
- should freeze at this point
- (if no <span class="productname">WinDbg</span> is waiting in the other
- virtual machine):</p>
-
- @{[ doc_img 'ntdebug-wait','Side being kernel-debugged waiting for <span class="productname">WinDbg</span>' ]}
-
-
- <a name="law"><h2>Laws and Licensing Conditions</h2></a>
-
- <p>If you are an <span class="productname">authorized user</span> of
- <span class="productname">Microsoft Windows NT</span> the laws in some
- countries give you the right to fully handle the product in any way you
- want. Therefore you can disassemble the product even in the case you had
- to agree with the product license forbidding such disassembly as the
- country laws override any such license agreement.</p>
-
- <h3>Microsoft Service Pack</h3>
-
- <p>Sometimes you may have the legal license for
- <span class="productname">Microsoft Windows NT</span>
- but for various technical reasons you do not have the media and/or
- installation ready at the place of intended use of this project.</p>
-
- <p>Fortunately <span class="productname">Microsoft</span> provides
- $freebeer update packages for its
- <span class="productname">Microsoft Windows</span> products called
- <span class="productname">Service Packs</span>; the latest one is
- <span class="productname">@{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp','Microsoft Windows XP Service Pack 1a' ]}</span>.</p>
-
- <p>This downloadable file contains the full versions of the essential
- files needed for the current stage of this product:
- <span class="fname">ntfs.sys</span>
- and
- <span class="fname">ntoskrnl.exe</span>.
- It even contains
- <span class="fname">cdfs.sys</span> and
- <span class="fname">fastfat.sys</span> for testing purposes.</p>
-
- <p><span class="productname">Service Pack</span> also contains
- EULA (End User License Agreement) paper disallowing any use of
- <span class="productname">Service Pack</span> outside its original
- intentions. According to the laws of some countries you need to be
- <span class="productname">authorized user</span> of the
- <span class="productname">Microsoft Windows XP</span> product to be
- allowed to use the files contained in such
- <span class="productname">Service Pack</span> without the bindings of its
- EULA. Even the interpretation of such laws may vary.</p>
-
- <p>It would be a breach of the law by the project author to provide
- automatic (=hidden) functionality to download and extract the
- <span class="productname">Service Pack</span> files. On the other hand it
- is perfectly legal to ask user for his/her confirmation whether he/she is
- really the <span class="productname">authorized user</span> of
- <span class="productname">Microsoft Windows XP</span> product and
- download/extract the <span class="productname">Service Pack</span> files
- accordingly.</p>
-
- @{[ doc_img 'captive-install-acquire-ask','Microsoft Windows Drivers Acquire Affirmation' ]}
-
- <h2>Project Architecture</h2>
-
- @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}
-
- <p>Most of the work of this project is located in the single box called
- "<span class="constant">libcaptive</span>" located in the center
- of the scheme. This component implements the core W32 kernel API by
- various methods described in this document.
- The "<span class="constant">libcaptive</span>" box cannot be
- further dissected as it is just an implementation of a set of
- @{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym','API functions' ]}.
- It could be separated to several subsystems such as the
- @{[ a_href '#cache_manager','Cache Manager' ]},
- Memory Manager, Object Manager, Runtime Library, I/O Manager
- etc. but they have no interesting referencing structure.</p>
-
- <p>As this project is in fact just a filesystem implementation every
- story must begin at the device file and end at the filesystem operations
- interface. The unified suppported interfaces are
- <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
- (the most low level portability, data-types and utility library for Gnome)
- <span class="type">GIOChannel</span> (for the device access) and the custom
- <span class="constant">libcaptive</span> filesystem API. Each of these ends
- can be connected either to some direct interface (such as the
- <span class="constant">captive-cmdline</span> client),
- @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
- or as a general $GnomeVFS filter.
- @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
- most cases as it offers standard filesystem interface by Linux kernel.
-
- You can also use $GnomeVFS as it offers nice filter interface on
- the UNIX user-privileges level for transparent operation with archives and
- network protocols. This filter interface was used by this project to turn
- the device reference such as <span class="fname">/dev/hda3</span> or <span
- class="fname">/dev/discs/disc0/part3</span> to the fully accessible
- filesystem (pretending being an "archive" in the device
- reference). This device access can be specified by $GnomeVFS URLs such as:
- <span
- class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
-
- <span class="constant">captive-bug-replay</span> serves just for debugging
- purposes — you can 'replay' existing
- <span class="fname">file.captivebug.xml.gz</span> automatically being
- generated during W32 filesystem failure. This bugreport file will contain
- all the touched data blocks of the device used in the moment of the
- failure. <span class="constant">captive-bug-replay</span> will therefore
- emulate internal virtual writable device out of these bugreported data.
-
- <p>If the passed device reference is requested by the user to be accessed
- either in <span class="dashdash">--ro</span> (read-only) mode or in the
- <span class="dashdash">--rw</span> (full read-write) mode there are no
- further device layers needed. Just in the case of <span
- class="dashdash">--blind</span> mode another layer is involved to emulate
- read-write device on top of the real read-only device by the method of
- non-persistent memory buffering of all the possible write requests.</p>
-
- <span class="constant">sandbox commit buffer</span> is involved only in the
- case @{[ a_href '#sandbox','sandboxing feature' ]} is active. It will
- buffer any writes to the device during the sandbox run to prevent
- filesystem damage if the driver would fail in the meantime. If the
- filesystem gets finally successfully unmounted this sandbox buffer can be
- <a name="safe_flush">safely flushed</a>
- to its underlying physical media. The buffer will be dropped
- in the case of filesystem failure, of course. The filesystem should be
- unmounted from time to time — it can be transparently unmounted and mounted
- by <span class="command">commit</span> of
- <span class="constant">captive-cmdline</span> custom client. Currently you
- cannot force remounting when using
- @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
- but it will be remounted after approx each 1MB data written automatically
- due to @{[ a_href '#log_file_full','NTFS log file full' ]}.
-
- Now we need to transparently
- @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','connect' ]}
- the device interface of <span class="type">GIOChannel</span> type through
- @{[ a_href '#sandbox','CORBA/ORBit' ]} to the sandboxed slave.
-
- <p>Such device is still only a UNIX style GLib <span
- class="type">GIOChannel</span> type at this point. As we need to supply it
- to the W32 filesystem driver we must convert it to the W32 I/O Device
- with its capability of handling <span class="type">IRP</span>
- (<span class="constant">I/O Request Packet</span>; structure holding the
- request and result data for any W32 filesystem or W32 block device
- operation)
- requests from its upper W32 filesystem driver. Such W32 I/O Device can
- represent either <span class="type">CD-ROM</span> or
- <span class="type">disk</span> device type as different W32 filesystem
- drivers require different media types — currently only
- <span class="fname">cdfs.sys</span> requires
- <span class="type">CD-ROM</span> type.</p>
-
- <p>W32 media I/O Device is accessed from the W32 filesystem driver.
- The filesystem driver itself always creates volume object by
- <span class="function">IoCreateStreamFileObject()</span> representing the
- underlying W32 media I/O Device as the object handled by the
- filesystem driver itself. All the client application filesystem requests
- must be first resolved at the filesystem structures level, passed to the
- volume stream object of the same filesystem and then finally passed to the
- W32 media I/O Device (already implemented by this project as an
- interface to <span class="type">GIOChannel</span> noted above).</p>
-
- <p>The filesystem driver is called by the core W32 kernel implementation of
- <span class="constant">libcaptive</span> in
- @{[ a_href '#synchronous','synchronous way' ]} in single-shot manner instead of
- the several reentrancies while waiting for the disk I/O completions as can
- be seen in the original
- <span class="productname">Microsoft Windows NT</span>.
- This single-shot synchronous behaviour is possible since all the needed
- resources (disk blocks etc.) can be always presented as instantly ready as
- their acquirement is solved by @{[ a_href 'hostosnote','Host-OS' ]} outside of
- the W32 emulated @{[ a_href 'guestosnote','Guest-OS' ]} environment.
- For several cases needed only by <span class="fname">ntfs.sys</span>
- there had to be supported asynchronous access — parallel execution
- is emulated by GLib <span class="function">g_idle_add_full()</span>
- with <span class="function">g_main_context_iteration()</span> called during
- <span class="function">KeWaitForSingleObject()</span>.</p>
-
- <p><span class="constant">libcaptive</span> offers the W32 kernel
- filesystem API to the upper layers. This is still not the API the common
- W32 applications are used to as they use W32 libraries which in turn pass
- the call to W32 kernel. For example
- <span class="function">CreateFileA()</span> is being implemented by several
- libraries such as <span class="fname">user32.dll</span> as a relay
- interface for the kernel function
- <span class="function">IoCreateFile()</span> implemented by this
- project's <span class="constant">libcaptive</span> W32 kernel
- emulation component.</p>
-
- <p>As it would be very inconvenient to use the legacy, bloated and UNIX
- style unfriendly W32 kernel filesystem API this project offers its own
- @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
- the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
- This interface is supposed to be easily utilized by
- <a href="#client_interface_customapp">a custom application accessing
- the W32 filesystem driver</a>.</p>
-
- <p>@{[ a_href '#sandbox','CORBA/ORBit' ]} hits us again – we need to
- @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','translate' ]}
- the @{[ a_href '#client_interface','custom filesystem API interface' ]}
- out of the sandboxed slave to the UNIX space.</p>
-
- <p><span class="constant">captive sandbox master</span> provides the
- functionality of covering any possible sandboxed slave restarts and its
- communication. It is also capable of
- <a name="demultiplexing_master">demultiplexing single API operations</a>
- to multiple its connected sandbox slaves in transparent way
- as each of them handles
- @{[ a_href '#mounted_one','just one filesystem device' ]}.</p>
-
- <p>The rest of the story is not much special for this project since this is
- a common UNIX problem how to offer user space implemented UNIX filesystem
- as a generic system filesystem (as those are usually implemented only as
- the components od UNIX kernel).</p>
-
- <p>The filesystem service can be offered in several ways:</p>
-
- <dl>
- <dt>Custom client</dt>
- <dd>
- <p>One possibility would be to write
- <a name="client_interface_customapp">a custom client application</a>
- for this project such as file manager or a shell. Although it
- would implement the most appropriate user interface to the set of
- functions offered by this project (and W32 filesystem API) it has the
- disadvantage of special client software. Appropriate client is provided
- by this project as:
- <span class="fname">src/client/cmdline/cmdline-captive</span></p>
- </dd>
-
- <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
- <dd>
- <p>The most usable interface is the
- @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
- by <span class="constant">liblufs-captivefs</span>.
- As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
- already assigns separate process for each filesystem mount the
- @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
- is not utilized in this case.</p>
-
- <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
- needs multiple operating threads (each UNIX kernel operation needs
- one free lufsd slot/thread to not to fail immediately).
- As <span class="constant">libcaptive</span> is
- @{[ a_href '#synchronous','single-threaded' ]} all the operations
- get always synchronized by
- <span class="constant">liblufs-captivefs</span>
- before their pass over to <span class="constant">libcaptive</span>.</p>
- </dd>
-
- <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
- <dd>
- <p>This client allowing its filesystem access even without any
- involvement of UNIX kernel from any $GnomeVFS aware client application
- (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
- This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
- data flow of this project in two points — both as the lowest layer
- device image source and also as the upper layer for the filesystem
- operation requests.</p>
- </dd>
- </dl>
-
- <p>Unimplemented and deprecated methods for providing filesystem
- service:</p>
-
- <dl>
- <dt>W32 filesystem in UNIX OS kernel</dt>
- <dd>
- <p>The real UNIX OS filesystem implementation must be completely
- implemented inside the hosting OS kernel. This requires special coding
- methods with limited availability of coding features and libraries.
- Also it would give the full system control to the untrusted W32
- filesystem driver code with possibly fatal consequences of yet
- unhandled W32 emulation code paths. It would benefit from the best
- execution performance but this solution was never considered a real
- possibility.</p>
- </dd>
-
- <dt>Custom NFS server</dt>
- <dd>
- <p>The common approach
- <a name="offered_NFS">of filesystem implementations</a>
- outside UNIX OS kernel were custom NFS servers usually running on the
- same machine as the NFS-connected client as such NFS server is usually
- an ordinary UNIX user space process. It would be possible to implement
- this project as a custom NFS server but the NFS protocol itself
- has a lot of fundamental flaws and complicated code for backward
- compatibility.</p>
- </dd>
- </dl>
-
-
- <a name="mounted_one"><h2>At Most One Mounted Filesystem</h2></a>
-
- <p>The project technically supports only one (exactly one...) mounted
- filesystem device and only one filesystem driver. There is nothing
- complicated to support multiple disks and multiple loaded filesystem
- modules but as they would share the address space it would only bring
- a possible complications during bug reports and the bug solving
- itself. It was considered as a more sane way to support multiple W32
- mounted disks by completely separately running project instances in
- a different UNIX processes communicating from their sandboxes via
- @{[ a_href '#sandbox','CORBA sandbox interface' ]}. This sandboxing
- feature is not yet deployed although its code is already prepared.</p>
-
- <p>The project also does not support any state cleanup to be able to load
- filesystem <span class="constant">A</span>,
- cleanup <span class="constant">A</span> and load a different
- filesystem <span class="constant">B</span> in the same process address
- space. It complies with the preventions of the possible debugging
- complications as noted above. Despite this you still must call the function
- <span class="function">captive_shutdown()</span> to flush all the pending
- filesystem buffers to the disk. After calling
- <span class="function">captive_shutdown()</span> the process address space is
- no longer usable for any further project operations and the process is
- expected to be terminated in the manner compatible with its driving
- @{[ a_href '#sandbox','CORBA sandbox interface' ]} control master.</p>
-
- <p>Each sandbox executing the untrusted W32 binary filesystem driver code
- is connected through its
- @{[ a_href '#sandbox','CORBA sandbox interface' ]} at the point of upper
- layer <span class="constant">libcaptive</span>-specific filesystem API, at
- the point of the bottom layer of <span class="type">GIOChannel</span>
- device access and also for transfers of GLib logging
- messages/warnings/errors out of the sandbox to the user.</p>
-
-
- <a name="sandbox"><h2>Sandboxing of W32 filesystem</h2></a>
-
- <p>The emulated W32 environment running the original W32 filesystem driver
- is separated from the rest of UNIX OS. It achieves the following goals:</p>
-
+ <li><a href="Reverse.html.pl#dumpbin"><span class="productname">dumpbin.exe</span></a></li>
+ <li><a href="Reverse.html.pl#WinDbg"><span class="productname">WinDbg</span> Windows NT kernel debugging</a>
+ <ul>
+ <li><a href="Reverse.html.pl#WinDbg_WinDbg"><span class="productname">WinDbg</span> side setup</a></li>
+ <li><a href="Reverse.html.pl#WinDbg_kern">Setup of the side being kernel-debugged</a></li>
+ </ul></li>
+ </ul></li>
+ </ul></li>
+
+<li><a href="Details.html.pl">Implementation Details</a>
+
+ <ul>
+ <li><a href="CacheManager.html.pl">NT Cache Manager</a>
<ul>
- <li><b>Restartable</b>: W32 driver can be restartde in clean state if it crashed</li>
- <li><b>Secure</b>: Malicious W32 code cannot affect the security of UNIX OS</li>
- <li><b>Stable</b>: Buggy W32 cannot crash any part of UNIX OS</li>
- </ul>
+ <li><a href="CacheManager.html.pl#TraceFS">TraceFS NT Cache Manager Tracer</a>
+ <ul>
+ <li><a href="CacheManager.html.pl#TraceFS_general">TraceFS for general API tracing</a></li>
+ </ul></li>
+ </ul></li>
- <p>Sandboxing is provided with the following attributes:</p>
-
+ <li><a href="Details.html.pl#emulmeth">Choice of the Emulation Methods</a>
<ul>
- <li>standalone UNIX process with separate memory space</li>
- <li>chroot(2) in empty directory to prevent any UNIX OS filesystem access</li>
- <li>setuid(2) to own user/group to prevent interaction with UNIX processes</li>
- <li>setrlimit(2) to limit system resources available for W32 environment</li>
- <li>the only connection with the UNIX OS by CORBA/ORBit RPC</li>
- </ul>
-
- <p>This security is almost the same as provided by
- emulated virtual machines such as
- @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware' ]}.</p>
-
- @{[ doc_img 'dia/inheritance','Sandboxing Scheme' ]}
-
- <p>Project can be also used in non-sandboxed mode by
- <span class="command">--no-sandbox</span> option as it is easier to debug
- without CORBA/ORBit RPC. In this case the
- <span class="type">DirectorySlave</span>/<span class="type">FileSlave</span>
- options are used directly instead of their
- <span class="type">DirectoryParent</span>/<span class="type">FileParent</span>
- peers.</p>
-
-
-<h1>Choice of the Emulation Methods</h1>
-
- <p>The intent of the project was to get reliable read-write access to
- <span class="productname">NTFS</span> partition. There are several possible
- ways to achieve that:</p>
-
- <h2>Virtualmachine Running the Original W32 Subsystem</h2>
-
- <p>Creating virtual-hardware PC and running the original W32 binaries
- including their boot-loader etc. Disk device access would be passed as
- virtual IDE disk (=hard disk drive). File access API would be implemented
- either by special escaping by some trapped instruction out of the
- virtualmachine while using W32 file access API or using the standard W32
- SMB (Server Message Block) network access through some virtual network
- card. The latter network access solution is almost the currently available
- possibility of running full-blown disk-sharing real
- <span class="productname">Microsoft Windows NT</span> inside virtual
- machine emulator such as <span class="productname">VMware</span>.</p>
-
- <p>pros: Full compatibility due to fully native codebase.</p>
-
- <p>cons: Hard to debug, missing documentation of NT booting internals,
- possible problems by different PC virtual-hardware than expected by NT,
- requirement of fully installed
- <span class="productname">Microsoft Windows NT</span> product.</p>
-
- <a name="method_ntoskrnl"><h2>"ntoskrnl.exe" Inside Virtual Address Space</h2></a>
-
- <p>This solution was chosen by the project. Binary filesystem driver and
- also <span class="fname">ntoskrnl.exe</span> binary file are required.
- Unfortunately <span class="fname">ntoskrnl.exe</span> expects a native
- PC virtual-hardware missing during regular UNIX user space process
- emulation, therefore such instructions must be trapped and emulated/ignored
- from case to case.</p>
-
- <p>Also the <a name="init_ntoskrnl">initialization code of <span
- class="fname">ntoskrnl.exe</span></a> is not executed by this project since
- it expects to get full PC hardware access privileges and thus some
- datastructures do not get initialized by it (need to be trapped later at
- runtime stage). Some of the missing initializations are solved by
- @{[ a_href '#functype_wrap','API functions wrapping' ]}.
-
- <p>pros: Lightweight, easier to debug.</p>
-
- <p>cons: Possible incompatible emulation of
- <span class="fname">ntoskrnl.exe</span> parts, missing documentation needed
- for the implementation.</p>
-
- <h2>Filesystem Driver Inside Virtual Address Space</h2>
-
- <p>Unlike @{[ a_href '#method_ntoskrnl','previous method' ]} here we do not use
- even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
- W32 is <a name="native_ntoskrnl">emulated from the project source
- files</a>. <span class="fname">cdfs.sys</span> driver was successfuly ran
- in this manner in the former versions of this project but the possibility
- to run without <span class="fname">ntoskrnl.exe</span> was dropped since it
- had no licensing gains (you need the original
- <span class="productname">Microsoft Windows NT</span> files at least for
- the filesystem driver itself) and the emulation of undocumented parts
- reusable from <span class="fname">ntoskrnl.exe</span> binary was
- a pain.</p>
-
- <p>pros: Lightweight, easier to debug.</p>
-
- <p>cons: Possible incompatible emulation of the whole
- <span class="fname">ntoskrnl.exe</span>, its missing documentation.</p>
-
-
-<h1>Implementation Details</h1>
-
- <a name="cache_manager"><h2>NT Cache Manager</h2></a>
-
- <p>Although there exist some 3rd party documents about
- <span class="productname">NT Cache Manager</span> W32 subsystem such as
- <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
- 'The NT Cache Manager Description' ]}</span> or
- <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
- 'Learn About NT'."'".'s File-system Cache' ]}</span>
- they are definitely insufficient for compatible
- <span class="productname">NT Cache Manager</span> reimplementation.</p>
-
- <p><span class="productname">NT Cache Manager</span> is about mapping
- filesystem objects such as regular file data, filesystem bitmap or
- journalling zone (log file). It is also being used by the filesystem for
- mapping of virtual volume files representing the whole underlying
- filesystem device.</p>
-
- <p>The original W32 <span class="productname">NT Cache Manager</span>
- is much more complicated as it must coordinate its effort with
- other W32 subsystems like mapping of executable files
- (<span class="type">ImageSectionObject</span>), insufficient system
- resources from <span class="productname">NT Memory Manager</span>
- or general effort to perform caching features for system performance.</p>
- <span class="productname">NT Cache Manager</span> of this project has much
- simpler goal - it just needs to provide compatible
- <span class="productname">NT Cache Manager</span> functionality while
- the other goals of its W32 counterpart are left to be successfuly handled
- by UNIX OS in much more efficient way.</p>
-
- @{[ doc_img 'dia/cache-manager',
- '<span class="productname">NT Cache Manager</span> Architecture' ]}
-
- <p>Cache Manager objects are always bound to
- <span class="type">FCB</span> (File Control Block).
- <span class="type">FileObject</span> (or its associated
- <span class="type">HANDLE</span>) serve only as reference
- to <span class="type">FCB</span> and there can be multiple
- <span class="type">FileObject</span>/<span class="type">HANDLE</span>
- items for one <span class="type">FCB</span>. It is a bit misleading
- you must use <span class="type">FileObject</span> pointer while calling
- most of the Cache Manager functions.</p>
-
- <p>Before using any other Cache Manager functions you must first call
- <span class="function">CcInitializeCacheMap()</span>. You must give the
- maximum mapped object offset. Each mapped object byte must have at most one
- mapped memory location - no shared pages are allowed. Also any subsequent
- mapping request is expected to be mapped into continuous memory region.
- It implies you must reserve the memory region for possible future mapping
- during the initial <span class="function">CcInitializeCacheMap()</span>
- moment sized according to the given maximum mapped object offset.
- This is the approach currently implemented by this project although it
- cannot be used for 3rd party <span class="fname">ext2fsd.sys</span>
- driver as it initialized Cache Manager by the whole media device size
- and it surprisingly succeeds for original
- <span class="productname">Microsoft Windows</span>
- <span class="productname">Cache Manager</span>.
- I expect the space reservation should be postponed to the first mapping
- request and expect no multiple mappings will be done in the case
- of memory-exceeding <span class="function">CcInitializeCacheMap()</span>
- reservation request. <span class="function">CcSetFileSizes()</span>
- changing the reserved memory area size may assume no existing Map
- or Pin mappings exist.</p>
-
- <p><span class="type">PCACHE_MANAGER_CALLBACKS</type> argument can be
- safely ignored:</p>
-
- <dl>
- <dt><span class="function">AcquireForReadAhead()</span>/<span class="function">ReleaseFromReadAhead()</span></dt>
- <dd>
- <p>As any readahead functionality is optional these entries are
- never used by Cache Manager implementation of this project.</p>
- </dd>
-
- <dt><span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span></dt>
- <dd>
- <p>Even the write-behind functionality is optional for Cache Manager.
- It is being done in asynchronous way in the original
- <span class="productname">Microsoft Windows</span>
- <span class="productname">Cache Manager</span>.
- implementation and it is ignored by Cache Manager implementation of
- this project.</p>
-
- <p>Cache Manager does not need to write any data if not explicitely
- requested by the driver. It is even expected to silently drop any
- pending dirty data blocks during filesystem shutdown.
- Forced dirty block write by function
- <span class="function">CcFlushCache()</span> should be written without
- any wrapping surrounding
- <span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span>
- pair.</p>
- </dd>
- </dl>
-
- <p><span class="function">CcUninitializeCacheMap()</span> is just
- a suggestion for Cache Manager that driver will no longer reference
- given <span class="type">SharedCacheMap</span>. The uninitialization
- can be postponed to any later moment in original
- <span class="productname">Microsoft Windows</span>
- <span class="productname">Cache Manager</span>
- as it may be locked by existing
- <span class="type">ImageSectionObject</span>
- of some file being executed etc.
- <a name="sharedcachemap_leak">It is fatal to destroy
- <span class="type">SharedCacheMap</span></a>
- in the moment you see no other
- references to it as the driver will access it for some moment
- even after <span class="function">CcUninitializeCacheMap()</span>.
- I am not sure if it is a bug of the driver or whether there are some rules
- how long after <span class="function">CcUninitializeCacheMap()</span>
- completion given <span class="type">SharedCacheMap</span> still exists.
- Fortunately it is safe to never destroy
- <span class="type">SharedCacheMap</span> and leave it leaked - everything
- gets clean in the
- @{[ a_href '#sandbox','sandboxed environment' ]} soon anyway.</p>
-
- <p>There exist Map and Pin type objects for each
- <span class="type">SharedCacheMap</span> although they look very similiar.
- Only these objects give you access to any memory data
- — <span class="type">SharedCacheMap</span> only reserved the space
- to ensure continuous mapping of the forthcoming mappings but it did not map
- any data into it.</p>
-
- <p>Mapping of 'new' Map or Pin will create the new object only in the case
- no such mapping exists now. Otherwise you will just get the reference to
- the existing object with increased usecount.</p>
-
- <dl>
- <dt>Map</dt>
- <dd>
- <p>Map mapping is always at most one for each
- <span class="type">SharedCacheMap</span>. Base offset/length of such
- mapping have no meaning as there can be only single Map.</p>
-
- <p>Apparently Map size can be arbitrary long according
- to its <span class="type">SharedCacheMap</span> reserved space.</p>
-
- <p>You cannot modify the memory mapped by Map in any way.
- As it is the same memory area (address) as the pages used by Pin
- objects you always access the last modified version by possible
- Pin of the same page.</p>
- </dd>
-
- <dt>Pin</dt>
- <dd>
- <p>Pin mapping always represents just one physical page
- (<span class="constant">PAGE_SIZE</span> &nspan; 4096 for i386).
- Its base offset/length can be safely extended to be aligned to the
- requested page.</p>
-
- <p>Pin can have associated pair of oldest and newest
- <span class="type>LSN</span> (Linear Sequence Number). It can be
- set by <span class="function">CcSetDirtyPinnedData()</span>
- and Cache Manager always tracks the lowest and highest
- reported <span class="type>LSN</span> for each page.
- <span class="type>LSN</span> is assumed to be
- <span class="constant">0</span> if not set.</p>
-
- <p>Any existing Pin mapping will be reused for further mappings
- as long as it is not ThreadOwned. In the moment you use
- <span class="function">CcSetBcbOwnerPointer()</span> you will detach
- the associated Pin pages from its
- <span class="type">SharedCacheMap</span>.
- Although they will further act as valid Pin mappings they will be no
- longer reused during new Pin mapping of the same page.
- There can exist multiple Pin mappings of the same page (although
- sharing the same memory space). This detaching must be implemented
- even in the
- @{[ a_href '#synchronous','single-threaded' ]} W32 implementation
- of this project as it is affecting the behaviour of Cache Manager.
- It was never
- @{[ a_href '#TraceFS','seen' ]} how to behave if multiple dirty Pin
- mappings of the same page exist.</p>
- </dd>
- </dl>
-
- <p>Only the pages not yet present in the memory must be read from the disk.
- You must not read any pages you do not need to as the driver does not
- expect it and it would corrupt its data buffers.</p>
-
- <p>Cache Manager of this project will destroy Pin or Map mappings after
- their last unreferencing (in opposite of
- @{[ a_href 'sharedcachemap_leak','leaked <span class="type">SharedCacheMap</span>' ]}).
- Despite it any dirty pages may still be held as the pages
- (including their <span class="type>LSN</span>s) are cached associated
- with <span class="type">SharedCacheMap</span>. It may be also possible
- original <span class="productname">Microsoft Windows</span>
- <span class="productname">Cache Manager</span>
- postpones Pin mapping destroy to later time but it does not matter.</p>
-
-
- <a name="TraceFS"><h3>TraceFS NT Cache Manager Tracer</h3></a>
-
- <p>@{[ a_href '#cache_manager','Cache Manager behaviour' ]} would be hard
- to analyze just by @{[ a_href '#reverse','reverse engineering' ]} as it
- is pretty complicated code cooperating with many other W32 kernel
- subsystems. It was chosen as easier way to trace it instead and validate
- all the Cache Manager assumptions by Cache Manager simulator.</p>
-
- @{[ doc_img 'dia/TraceFS','TraceFS Hooking' ]}
-
- <p>You must prepare your driver to be hooked
- (<span class="fname">ntfs.sys</span> in this case):</p>
-
- <blockquote class="command">
- <p>@{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} ntfs.sys ./src/TraceFS/TraceFS-W32/TraceFS.sys >hooked/ntfs.sys</p>
- </blockquote>
-
- <p>This <span class="fname">hooked/ntfs.sys</span> file must be replaced
- in the <span class="fname">%System32%\\drivers</span> directory.
- Beware as
- <span class="productname">Microsoft Windows</span>
- has many backups of these system files such as
- <span class="fname">%System32%\\dllcache</span> — delete them
- all!</p>
-
- <p>You also need to install
- <span class="fname">./src/TraceFS/TraceFS-W32/TraceFS.sys</span>
- into <span class="fname">%System32%\\drivers</span> directory
- and import <span class="fname">TraceFS/TraceFS-W32/TraceFS.reg</span>
- registry file to initialize the debug driver during system boot.</p>
-
- <p>You can now pray a bit and snap the resulting Cache Manager tracing
- from <span class="productname">WinDbg</span> by
- @{[ a_href '#WinDbg','W32 remote kernel debugging' ]}:</p>
-
- @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}
-
- <p>The resulting trace file should be processed by
- @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
- implementation to validate its assumptions about Cache Manager behaviour.
- Any seen incompatibilies will be reported — your target is to reach
- as few error messages as possible.</p>
-
- <p>KNOWN BUGS: Combination of message synchronization primitives and
- implemented refusal to create journalling thread of
- <span class="fname">ntfs.sys</span>
- causes fatal system lockup in several advanced operations
- such as setting compression attribute. Despite it more common operations
- can be successfuly traced during the whole
- <span class="productname">Microsoft Windows</span>
- session including its final shutdown and such traces provide enough
- material to be food to
- @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
- validator.</p>
-
- <h4>TraceFS for general API tracing</h4>
-
- <p>Although TraceFS was up to now used only for tracing of
- <span class="productname">NT Cache Manager</span> it can be easily
- used ever for any other NT kernel API tracing. You need to provide
- appropriate function wrappers in the main source file
- @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.c' ]}
- and you must also export them in
- @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.def' ]}.
- @{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} has no hardcoded
- function names – it will hook exactly the exported entries.</p>
-
- <p>Framework for thread synchronizations and debug tracing is provided to
- prevent mangling of messages while running by multiple threads at once.
- Testing was done just on uniprocessor machine, SMP kernel may need some
- fixes.</p>
-
-
- <a name="functype"><h2>API Function Implementation Choices</h2></a>
-
- <p>For each function exported by W32
- <span class="fname">ntoskrnl.exe</span> and imported and called by the
- filesystem driver a decision needs to be made to properly implement its
- functionality. Currently implemented functionality statistics are provided
- below:</p>
-
- <table border="1" align="center">
- <tr><th>Function type </th><th>Items</th><th>Portion</th></tr>
- <tr><td>@{[ a_href '#functype_pass','pass' ]} </td><td> 81</td><td> 26%</td></tr>
- <tr><td>@{[ a_href '#functype_wrap','wrap' ]} </td><td> 2</td><td> 0%</td></tr>
- <tr><td>@{[ a_href '#functype_native_reactos','native-ReactOS' ]}</td><td> 113</td><td> 36%</td></tr>
- <tr><td>@{[ a_href '#functype_native_libcaptive','native-own' ]} </td><td> 116</td><td> 38%</td></tr>
- <caption>Function Implementation Types Statistics</caption>
- </table>
-
- <p>As there are several choices to implement each function the usual
- attempts/investigations ordering is listed in the sections below.</p>
-
- <p>Special case must be taken for data-type symbols since they are
- referenced without the possibility of catching the code flow by some
- breakpoints (it would be possible only in some special access cases). Data
- export symbols of <span class="constant">unpatched</span> libraries must
- contain already prepared content at the runtime. There is a problem
- with <span class="constant">patched</span> libraries where it is necessary
- to also fully implement the data symbol as
- @{[ a_href '#functype_native','native implementation' ]} since there is no
- possibility to @{[ a_href '#functype_pass','pass' ]} the data symbol instead of
- the original W32 data location and therefore there will be two instances of
- such data variable place. As there will be also the uncaught references for
- such W32 data location from the <span class="constant">patched</span>
- library itself such symbols should be usually only some constants (such as
- <span class="constant">KeNumberProcessors</span>).</p>
-
- <p>W32 platform symbols export/import can be based either on the symbol
- name itself or it can be also exported and imported just by its
- identification number called <span class="constant">Ordinal</span>.
- Although it saves some jumptables file binary size it is currently no
- longer used by W32 binaries and this project also does not support such
- <span class="constant">Ordinal</span> symbol reference type at all.</p>
-
- <p>All the exporting magic is handled by custom script
- <span class="fname">captivesym</span> processing the definition file
- <span class="fname">@{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym' ]}</span>
- to produce the intermediate relaying code
- <span class="fname">src/libcaptive/ke/exports.c</span>. For details of the
- <span class="fname">captivesym</span>-specific source file syntax please
- see its documentation:
- <span class="fname">@{[ a_href
- $W->{"top_dir"}.'/project/Pod2Html.html.pl?cvs=priv/captive/src/libcaptive/ke/captivesym.pl',
- 'src/libcaptive/ke/captivesym.pl' ]}</span>
-
- <a name="functype_pass"><h3>Direct Pass to Original "ntoskrnl.exe"</h3></a>
-
- <p>Simple (standalone) functions such as
- <span class="function">RtlTimeToSecondsSince1970()</span> can be simply
- passed to the original implementation in
- <span class="fname">ntoskrnl.exe</span> as they make no hardware access
- and they do not expect any special internal data structures to be set up
- in advance by an earlier library initialization. A common case are all
- the data structures utility functions such as
- <span class="constant">GenericTable</span> subsystem or
- <span class="constant">LargeMcb</span> handling.</p>
-
- <a name="functype_pass_fromunix"><h4>Pass from UNIX Code</h4></a>
-
- <p>Control flow begins in some standard UNIX code. Such code is always
- using @{[ a_href '#calltype_cdecl','cdecl call type' ]} for all its
- intracalls. <a href="#functype_native_reactos">Native functions
- compiled from <span class="productname">ReactOS</span> sources</a> use
- their own @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} declarations
- but these call type modifications are discarded during compilation for
- this project by the <span class="constant">LIBCAPTIVE</span>
- symbol.</p>
-
- <p>UNIX code calls <span class="function">FUNCTIONNAME()</span> relay
- from the generated UNIX jump table. Such relay will debug dump the
- passed arguments and finally pass the control to the original W32
- function code in the proper call type
- @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} for a given
- function.</p>
-
- <p>Original W32 code entry point is always trapped by a breakpoint
- although it would not be needed during this specific direct pass from
- UNIX code to the original W32 implementation. Still the breakpoint has
- to be there to catch some other (such as intra-W32) possible calls
- described later. There are several more ways to define breakpoint in
- the code. One way is to use processor hardware breakpoint support but
- the number of breakpoints is limited. The other way is to patch in the
- <span class="instruction">@{[ 'int $3' ]}</span> instruction but it will invoke
- <span class="constant">SIGTRAP</span> signal handler conflicting with
- the possible debugger (<span class="productname">gdb(1)</span>)
- control. This project uses the <span class="instruction">hlt</span>
- instruction, which also has a single-byte opcode as
- <span class="instruction">@{[ 'int $3' ]}</span> and it is a privileged
- instruction forbidden to be used from the UNIX user space code.
- <span class="instruction">hlt</span> invokes
- <span class="constant">SIGSEGV</span> signal which can be resolved by
- a custom signal handler without any conflict with the possible
- debugger control; <span class="productname">gdb(1)</span> needs the
- following command to pass through such
- <span class="constant">SIGSEGV</span> signal:</p>
-
- <blockquote class="command">
- <p>handle SIGSEGV nostop noprint pass</p>
- </blockquote>
-
- <p>When a breakpoint gets caught, we usually need to return to the
- running code. Unfortunately it is not possible because of the patched
- breakpoint opcode. The breakpoint cannot be simply removed upon return
- as it would permanently loose control over the point of entry. Even if
- the return would include faking of the return address in the bottom
- stack frame to patch the breakpoint back during later function exit it
- still would not solve the caughts of inner calls of recursive
- functions. One of the working possibilities would be to patch the
- original instruction back and perform a singlestep provided by
- <span class="function">ptrace(2)</span> syscall. However such
- singlestep needs another controlling UNIX process and it would again
- conflict with the debuggers such as
- <span class="productname">gdb(1)</span>. This project implements the
- singlestep functionality by two consecutive breakpoints
- (<span class="instruction">hlt</span> instructions to be specific):
- The first two instruction addresses of the W32 functions are called
- <span class="productname">slot #1</span> and
- <span class="productname">slot #2</span>, the length of the first
- function instruction has to be analyzed to get the right address of
- <span class="productname">slot #2</span>. When the first breakpoint is
- caught it is necessary to patch the original instruction back and also
- patch another breakpoint in place of
- <span class="productname">slot #2</span>.
- During the <span class="productname">slot #2</span> breakpoint
- invocation the operation will be reverted — the breakpoint will be put
- to <span class="productname">slot #1</span> again and the instruction
- of <span class="productname">slot #2</span> will be restored to be able
- to continue the execution of the function.</p>
-
- <p>W32 function will finish in its specific
- @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}, the control
- will return to the UNIX jump table relay which will debug dump the
- return value and it will finally pass the control back to the UNIX
- caller in the standard UNIX
- @{[ a_href '#calltype_cdecl','cdecl call type' ]}.</p>
-
- @{[ doc_img 'fig/functype_patched_pass_fromunix',
- 'Function Type: <span class="constant">pass</span> from UNIX Code' ]}
-
- <a name="functype_pass_fromw32"><h4>Pass from W32 Code</h4></a>
-
- <p>This function type is similiar to the
- @{[ a_href '#functype_pass_fromunix','previous one' ]} with the exception
- of more complicated entry point. Unfortunately W32 libraries call their
- own functions directly, using the <span class="instruction">call</span>
- instructions without any patchable jump table. Even the
- <span class="instruction">call</span> argument itself cannot be patched
- according to the relocation table record as such library intra-call
- instruction has no relocation due to its relative argument offset on
- <span class="constant">i386</span>. This time the double-breakpoint
- mechanism @{[ a_href '#functype_pass_fromunix','described above' ]} gets
- handy since it will catch the entry point when the function gets
- called. <span class="constant">SIGSEGV</span> handler gets invoked by
- the <span class="instruction">hlt</span> instruction and it will
- redirect the control to the jump table relay function to debug dump the
- function entry arguments (it has no other uses in this call type).</p>
-
- <p>When the relay needs to call the original function it will reach
- exactly the same breakpoint instruction as during the recent
- <span class="constant">SIGSEGV</span> handling redirecting to this
- calling relay. But this time the
- <span class="constant">through_w32_func</span> field of this function
- record will be set to to prevent repeated redirection and to pass the
- control through the breakpoint mangle instead this time.</p>
-
- <p>Returning is not much interesting as the first
- <span class="constant">SIGSEGV</span> handler did a straight jump
- for the redirection purposes without any needed consequent
- handling.</p>
-
- <p>The jump table relay used for the callers from W32 code is
- a different one than the relay being used for the callers
- @{[ a_href '#functype_pass_fromunix','from UNIX code' ]}. UNIX code always
- uses relay with external @{[ a_href '#calltype_cdecl','cdecl call type' ]}
- but in this case a relay with the appropriate
- @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} is used.</p>
-
- @{[ doc_img 'fig/functype_patched_pass_fromw32',
- 'Function Type: <span class="constant">pass</span> from W32 Code' ]}
-
- @{[ vskip() ]}
-
- <table border="1" align="center">
- <tr><td><span class="fname">captivesym</span> keyword</td><td>pass</td></tr>
- <tr><td>Native code function name </td><td>(no implementation)</td></tr>
- <tr><td>W32 traced code from UNIX function name </td><td>FUNCNAME</td></tr>
- <tr><td>W32 traced code from W32 function name </td><td>FUNCNAME_cdecl/_stdcall/_fastcall</td></tr>
- <tr><td>Entry/exit debug tracing from UNIX code </td><td>yes</td></tr>
- <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
- <caption>Function Type <span class="constant">pass</span> Characteristics</caption>
- </table>
-
- <a name="functype_wrap"><h3>Wrap of the Original "ntoskrnl.exe" Function</h3></a>
-
- <a name="functype_wrap_fromunix"><h4>Wrapping of Call from UNIX Code</h4></a>
-
- <p>The code control flow has no special hardcore features since it is
- very similiar to <a href="#functype_pass_fromunix">the direct pass to
- W32 function from UNIX code</a>. All the wrapping is done in the
- standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.
- Jump table debug dumping relays are provided twice — the
- "outer" one to trace the parameters from the function caller
- and the "inner" one to trace the call from the wrapper to the
- original W32 code. The "inner" relay also calls the W32 code
- with the appropriate <a href="#calltype">cdecl/stdcall/fastcall call
- type</a>.</p>
-
- @{[ doc_img 'fig/functype_patched_wrap_fromunix',
- 'Function Type: <span class="constant">wrap</span> from UNIX Code' ]}
-
- <a name="functype_wrap_fromw32"><h4>Wrapping of Call from W32 Code</h4></a>
-
- <p>This scheme is a combination of the
- <a href="#functype_wrap_fromunix">previous wrap of a call from
- UNIX code</a> and the <a href="#functype_pass_fromw32">direct pass from
- the W32 code</a>. The control is caught and redirected by
- <span class="constant">SIGSEGV</span> handler from the breakpoint
- placed at the entry to the original W32 function code. The second entry
- to the original W32 function with the
- <span class="constant">through_w32_func</span> field of this function
- description already set is done from the "inner" jump table
- relay with the appropriate
- @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}.</p>
-
- @{[ doc_img 'fig/functype_patched_wrap_fromw32',
- 'Function Type: <span class="constant">wrap</span> from W32 Code' ]}
-
- @{[ vskip() ]}
-
- <p>Some functions can be <a href="#functype_pass">passed to the original
- code</a> but they need their parameters to be checked/prepared.
- Currently, such wrapping is only needed for the
- <span class="function">ExAllocateFromPagedLookasideList()</span> function
- where it is required due to <a href="#init_ntoskrnl">missing execution of
- <span class="fname">ntoskrnl.exe</span> initialization execution</a>,
- which would otherwise properly initialize some internal data structures.
- In this case the wrapping code detects passing of an uninitialized
- parameter and will search through the whole
- <span class="fname">ntoskrnl.exe</span> code body at runtime to find the
- proper initialization routine containing the correct initialization
- parameters. Passed addresses of static structures must be differentiated
- as each of them usually has different initialization parameters. It is
- proactive to not to have fixed parameters array as these parameters may
- differ across different <span class="fname">ntoskrnl.exe</span>
- versions.</p>
-
- <table border="1" align="center">
- <tr><td><span class="fname">captivesym</span> keyword</td><td>wrap</td></tr>
- <tr><td>Native UNIX wrapping code function name </td><td>FUNCNAME_wrap</td></tr>
- <tr><td>W32 traced wraping code from UNIX func. name </td><td>FUNCNAME</td></tr>
- <tr><td>W32 traced wrapping code from W32 func. name </td><td>FUNCNAME_cdecl/_stdcall/...</td></tr>
- <tr><td>W32 traced original code function name </td><td>FUNCNAME_orig</td></tr>
- <tr><td>Entry/exit debug tracing from UNIX code </td><td>yes</td></tr>
- <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
- <caption>Function Type <span class="constant">wrap</span> Characteristics</caption>
- </table>
-
- <a name="functype_native"><h3>Native Implementation</h3></a>
-
- <h4>Native Implementation Called from UNIX Code</h4>
-
- <p>This is the simplest case of a function call as it is fully
- handled only by the compiler and/or linker.</p>
-
- <p>In this case though, no debug dumping call relay is provided — such
- relay would need to rename the implementations of native functions to
- prevent its automatic linking with the caller code. This renaming would
- not be possible to do by simple <span class="constant">#define</span>
- since it would also rename any calling statements of such function in
- the same C sources. One of the possibilities to solve would be to
- utilize <span class="dashdash">--redefine-sym</span> feature of the
- <span class="productname">objcopy(1)</span> utility. On the other hand
- there is not much need to catch/debug such calls as both the caller and
- the callee are provided with full source file debug information for the
- debugger. Also the callee usually debug dumps its entry/exit parameters
- by custom debug dumps in the
- <a href="#functype_native_reactos"><span class="productname">ReactOS</span> implementations</a>.
-
- @{[ doc_img 'fig/functype_native_fromunix',
- 'Function Type: <span class="constant">native</span> from UNIX Code' ]}
-
- <a name="functype_native_fromw32"><h4>Native Implementation of
- "unpatched" Library Function Called from W32 Code</h4></a>
-
- @{[ doc_img 'fig/functype_unpatched_native_fromw32',
- 'Function Type: <span class="constant">native</span> of <span class="constant">unpatched</span> from W32 Code' ]}
-
- <p>Here comes the differentiation if the project deals either with
- a <span class="constant">patched</span> or an
- <span class="constant">unpatched</span> version of the library
- (<span class="constant">patched</span> is a loaded W32 binary
- library while <span class="constant">unpatched</span> library is
- completely provided by this project with no use of the library's
- original W32 binary file). As the project adjusts the exported symbol
- address during the patching operation, in some cases the
- <span class="constant">patched</span> library call may be handled
- simply as <span class="constant">unpatched</span> library call even for
- the <span class="constant">patched</span> libraries. Fortunately the
- distinction is not much important as the project is prepared to
- properly handle both cases.</p>
-
- <p>The W32 caller which imported the symbol will be pointed right to
- the relaying function. The debug dumping relay will be called from W32
- code with the appropriate
- @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} while the
- relay will call the implementation of the native function in the
- standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.</p>
-
- <h4>Native Implementation of "patched" Library Function Called from W32 Code</h4>
-
- @{[ doc_img 'fig/functype_patched_native_fromw32',
- 'Function Type: <span class="constant">native</span> of <span class="constant">patched</span> from W32 Code' ]}
-
- <p>The calling scheme is similiar to the
- <a href="#functype_native_fromw32">previous call of
- <span class="constant">unpatched</span> library function from W32
- code</a> but the call control is redirected from the entry point of the
- original W32 binary implementation by the breakpoint and its
- <span class="constant">SIGSEGV</span> handler as in
- <a href="#functype_pass_fromw32">the case of passing control from W32
- call</a>.</p>
-
- <p>The original W32 function implementation located in the original
- loaded binary file is never executed but its entry point needs to be
- trapped by the breakpoint to be able to catch the function calls within
- the library.</p>
-
- @{[ vskip() ]}
-
- <p>In all cases the final function implementation is a standard UNIX
- code compiled from C sources with full debug information available
- for the debugger. Fortunately all such functions do not need to be coded
- from scratch for this project since there already exist $freespeech
- $ReactOS and $Wine projects and their code can be used instead.</p>
-
- <p>$Wine project is listed mostly for a completeness as almost no
- code was suitable for reuse as it implements W32 user space while this
- project is running pure W32 kernel space environment (in $gnulinux user
- space!).</p>
-
- <a name="functype_native_reactos"><h4>Native Implementation
- - <span class="productname">ReactOS</span></h4></a>
-
- <p>Some functions are already implemented in the $ReactOS
- project and they can be used as they are. Although it would be
- possible to <a href="#functype_pass">pass some function calls to the
- original code</a> it is more handy to provide native implementation as
- there is better control of the data handling during debugging sessions
- due to the provided debugging symbols.</p>
-
- <p>Such functions can be found in
- <span class="fname">src/libcaptive/reactos/</span> subdirectory.
- Some functions had to be adjusted for this project
- - these modifications are compiled conditionally, depending on the
- <span class="constant">LIBCAPTIVE</span> symbol existence.</p>
-
- <p>Later stages of this project reached the level where
- $ReactOS is yet too immature and the needed functions are usually
- written just with the sad body:</p>
-
- <blockquote class="command">
- <p>UNIMPLEMENTED;</p>
- </blockquote>
-
- <p>Functions that were not possible to
- @{[ a_href '#functype_pass','pass' ]} were reimplemented by this project
- and placed in the project's implementation directories
- @{[ a_href '#reactos_nocare','instead of extending' ]} $ReactOS code.</p>
-
- <a name="functype_native_wine"><h4>Native Implementation – <span class="productname">Wine</span></h4></a>
-
- <p>Even though $Wine only implements the
- <span class="productname">Microsoft Windows NT</span> user space, there
- still are some common functions which could be copied from the $Wine
- project.</p>
-
- <a name="functype_native_libcaptive"><h4>Native Implementation – Project Specific</h4></a>
-
- <p>As the last resort it was necessary to provide completely own
- implementation of some API functions such as PC hardware dependent
- parts or memory management functions.</p>
-
- @{[ vskip() ]}
-
- <table border="1" align="center">
- <tr><td><span class="fname">captivesym</span> keyword</td><td>(none; just the symbol name)</td></tr>
- <tr><td>Native code function name </td><td>FUNCTIONNAME</td></tr>
- <tr><td>Native traced code from W32 code func. name </td><td>FUNCTIONNAME_cdecl/_std...</td></tr>
- <tr><td>Entry/exit debug tracing from UNIX code </td><td>no</td></tr>
- <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
- <caption>Function Type <span class="constant">native</span> Characteristics</caption>
- </table>
-
- <a name="functype_undef"><h3>Undefined Function</h3></a>
-
- <p>Functions not defined by any of the previous function types cannot be
- called by any W32 code including the code of the library implementing
- such function. All functions of <span class="constant">patch</span>ed
- libraries not listed in the <span class="fname">captivesym</span> exports
- file are automatically set to be trapped as fatal program execution
- errors.</p>
-
- <p>It is not necessary to list the symbols as
- <span class="constant">undef</span> as long as you are just loading the
- W32 <span class="constant">PE-32</span> code and the symbols belong to
- <span class="constant">patch</span>ed library. On the other hand if you
- are loading W32 <span class="fname">.so</span> code or if such symbol is
- a part of <span class="constant">unpatched</span> library (and thus
- being completely provided by the project) you need to list such symbol as
- <span class="constant">undef</span> type to prevent unresolved symbol
- reference.</p>
-
- <table border="1" align="center">
- <tr><td><span class="fname">captivesym</span> keyword</td><td>undef</td></tr>
- <tr><td>Native code function name </td><td>(no implementation)</td></tr>
- <tr><td>Native traced code function name </td><td>FUNCTIONNAME_cdecl/_stdcall/_fastcall</td></tr>
- <tr><td>Debug tracing message from UNIX code </td><td>yes</td></tr>
- <tr><td>Debug tracing message from W32 code </td><td>yes</td></tr>
- <caption>Function Type <span class="constant">undef</span> Characteristics</caption>
- </table>
-
-
- <a name="calltype"><h2>API Function Calling Conventions</h2></a>
-
- <p>Standard UNIX code compiled by GCC (GNU C Compiler) running on host
- $gnulinux always uses @{[ a_href '#calltype_cdecl','cdecl' ]} ABI (Application
- Binary Interface) calling convention. This calling convention is also the
- default declaration type of UNIX functions.</p>
-
- <p>W32 uses three different calling conventions in its ABI. They are all
- described in the
- <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclang/html/_core_argument_passing_and_naming_conventions.asp"><span class="productname">Microsoft</span> documentation</a>.
- There is always necessary to have the proper function declaration
- (prototype) in the caller scope to prevent all sorts of unexpected
- crashes.</p>
-
- <p>Unfortunately some non-matching combinations of calling conventions
- result in hard to debug bugs: the caller gets back an unexpected stack
- pointer from the callee and upon return it will restore registers from the
- wrong stack pointer place. Since the caller will finally reclaim its stack
- frame from its (uncorrupted) <span class="constant">EBP</span> stack frame
- pointer the caller will return to the caller of the caller correctly. Just
- the registers remain corrupted causing crashes of completely unrelated code
- executed far, far away...</p>
-
- <p><span class="constant">EDI</span>, <span class="constant">ESI</span> and
- <span class="constant">EBX</span> registers are always saved on the stack.
- They are stored on the stack in this particular order from bottom to top
- addresses (using the <span class="instruction">push EBX</span>,
- <span class="instruction">push ESI</span>,
- <span class="instruction">push EDI</span> sequence). Fortunately $gnulinux
- GCC has the same register saving behaviour. If some register corruption
- occurs the calling type presented between the caller and callee should be
- checked.</p>
-
- <a name="calltype_cdecl"><h3>W32 Calling Convention "cdecl"</h3></a>
-
- <p>The only calling convention in the UNIX world. The default one for all
- the compilers. All the arguments are passed on the stack, no arguments
- are cleaned by the callee. Possible inconsistencies in the number of
- function arguments with the function prototype used by the caller is
- harmless. Variable arguments lists can be passed by this convention.</p>
-
- @{[ doc_img 'fig/calltype_cdecl',
- 'W32 Calling Convention <span class="constant">cdecl</span> Scheme' ]}
-
- <table border="1" align="center">
- <tr><td>Arguments freed by </td><td>caller</td></tr>
- <tr><td>Arguments on the stack </td><td>#0 ... #(n-1)</td></tr>
- <tr><td>Arguments in the registers </td><td>none</td></tr>
- <tr><td>GCC attribute </td><td><span class="command">__attribute__((__cdecl__))</span> (default)</td></tr>
- <caption>Calling Convention <span class="constant">cdecl</span> Characteristics</caption>
- </table>
-
- <h3>W32 Calling Convention "stdcall"</h3>
-
- @{[ doc_img 'fig/calltype_stdcall',
- 'W32 Calling Convention <span class="constant">stdcall</span> Scheme' ]}
-
- <p>Convention never used in the UNIX world. It needs to be specified for
- W32 compilers. All the arguments are passed on the stack, all the
- arguments are cleaned by the callee. Possible inconsistencies in the
- number of function arguments with the function prototype used by the
- caller will result in fatal crash. Variable arguments lists cannot be
- passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
- instead.</p>
-
- <table border="1" align="center">
- <tr><td>Arguments freed by </td><td>callee</td></tr>
- <tr><td>Arguments on the stack </td><td>#0 ... #(n-1)</td></tr>
- <tr><td>Arguments in the registers </td><td>none</td></tr>
- <tr><td>GCC attribute </td><td><span class="command">__attribute__((__stdcall__))</span></td></tr>
- <caption>Calling Convention <span class="constant">stdcall</span> Characteristics</caption>
- </table>
-
- <h3>W32 Calling Convention "fastcall"</h3>
-
- <p>Convention never used in the UNIX world. It needs to be specified for
- W32 compilers. Convention used in the W32 world for its low calling
- overhead. All but the first two arguments are passed on the stack, such
- arguments are cleaned by the callee. First two arguments are passed in
- the registers <span class="constant">ECX</span> and
- <span class="constant">EDX</span> respectively. Possible inconsistencies
- in the number of function arguments with the function prototype used by
- the caller will result in fatal crash. Variable arguments lists cannot be
- passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
- instead.</p>
-
- <p>GCC (GNU C Compiler) native support for this calling convention
- is pretty fresh and it is currently present only in the recent CVS
- versions since 21st December of 2002 which should get released as GCC
- version 3.4. This project solved the unsupported calling convention by
- declaration of arguments passed in registers by
- <span class="command">__attribute__((__regparm__(3)))</span>.
- W32 passes the arguments in registers in the order
- <span class="constant">ECX</span>, <span class="constant">EDX</span> but
- GCC passes them in registers <span class="constant">EAX</span>,
- <span class="constant">EDX</span>, <span class="constant">ECX</span>.
- This incompatibility is compensated at C source level in the
- @{[ a_href '#functype','relaying code' ]} generated by
- <span class="fname">captivesym</span> relay generator.</p>
-
- @{[ doc_img 'fig/calltype_fastcall',
- 'W32 Calling Convention <span class="constant">fastcall</span> Scheme' ]}
-
- <table border="1" align="center">
- <tr><td>Arguments freed by </td><td>callee</td></tr>
- <tr><td>Arguments on the stack </td><td>#2 ... #(n-1)</td></tr>
- <tr><td>Arguments in the registers </td><td><span class="constant">ECX</span>=#0,
- <span class="constant">EDX</span>=#1</td></tr>
- <tr><td>GCC ≥3.4 attribute </td><td><span class="command">__attribute__((__fastcall__))</span></td></tr>
- <tr><td>GCC <3.4 attr. emulation</td><td><span class="command">__attribute__((__stdcall__))</span></td></tr>
- <tr><td> </td><td><span class="command">__attribute__((__regparm__(3) /* EAX,EDX,ECX */))</span></td></tr>
- <caption>Calling Convention <span class="constant">fastcall</span> Characteristics</caption>
- </table>
-
- <a name="synchronous"><h2>Multithreading and Multiple Processors</h2></a>
-
- <p>W32 platform stands on its thorough architecture parallelism. It
- must lock all its objects to maintain coherence in presence of
- multithreading and multiple processors. Since the author of this project
- considers any parallel execution a serious obstacle for debugging the whole
- project architecture was designed to prevent any undeterministic behaviour.
- Therefore this projects always emulates uniprocessor
- <span class="productname">Microsoft Windows NT</span> kernel
- (<span class="constant">KeNumberProcessors</span> symbol is always 1),
- everything runs in the single initial thread/process and all the filesystem
- operations are performed as synchronous
- ("synchronous" by flags
- <span class="constant">FILE_SYNCHRONOUS_IO_ALERT</span>,
- <span class="constant">FO_SYNCHRONOUS_IO</span>,
- <span class="constant">IRP_SYNCHRONOUS_API</span>,
- <span class="constant">IRP_SYNCHRONOUS_PAGING_IO</span>,
- forced <span class="constant">TRUE</span> result of
- <span class="function">IoIsOperationSynchronous()</span>
- etc.).
- For several cases needed only by <span class="fname">ntfs.sys</span> there
- had to be supported asynchronous access
- (<span class="constant">STATUS_PENDING</span> return code) – parallel
- execution is emulated by GLib
- <span class="function">g_idle_add_full()</span> with
- <span class="function">g_main_context_iteration()</span> called during
- <span class="function">KeWaitForSingleObject()</span>.</p>
- Since there is a possibility a real W32 parallel threading would
- be yet needed in the future all the code that would be hit by W32
- multithreading capability is marked by
- <span class="constant">TODO:thread</span> comment.</p>
-
- <p>Multiple processors (SMP) support will never need to be implemented
- since uniprocessor W32 kernels apparently run the filesystem driver modules
- fine. As this project implements only the uniprocessor W32 kernel all the
- processor locking functions and structures such as
- <span class="constant">KSPIN_LOCK</span> etc. can be safely implemented as
- no-operations.</p>
-
- <p>Asynchronous callbacks registered for
- <span class="constant">IO_WORKITEM</span>s are passed as GLib idle
- functions by <span class="function">g_idle_add_full()</span>. Although they
- will probably never be executed during non-interactive project's batch
- executions it is the responsibility of W32 driver implementation to
- complete all the pending tasks before its W32 shutdown. Such W32 shutdown
- is done during cleanup of the project's execution by
- <span class="function">captive_shutdown()</span>.</p>
-
- <a name="paranoia"><h2>Paranoia Checks</h2></a>
-
- <p>A general approach of software projects development is to implement
- many internal sanity checks during the development stage but to produce the
- most optimized final release product without those debugging checks.</p>
-
- <p>Facilities for these practices can be seen in the standard
- C include files for example as function
- <span class="function">assert()</span> which gets disabled by the
- <span class="constant">NDEBUG</span> symbol used during the final optimized
- executable compilation. This project uses Gnome GLib messaging subsystem
- offering sanity checks discarded by symbols
- <span class="constant">G_DISABLE_ASSERT</span> and
- <span class="constant">G_DISABLE_CHECKS</span>.
- <span class="productname">Microsoft</span> also produces two versions of
- its products – regular customers use the "free build" (also
- called "retail") while the programmers should develop their code
- on the "checked build" product releases.</p>
-
- <p>As this project will always run unknown binary code of proprietary W32
- filesystem drivers, the code can never be trusted. Such code even runs in
- the same unprotected address space as its controlling UNIX code. Since
- there is not enough documentation for the W32 components of the system and
- also such documentation is usually misleading it can never be considered as
- 100% emulation. Even in the final releases all the sanity checks
- implemented in this project should remain active as all the project's code
- always interacts with unknown and untrusted W32 binaries.</p>
-
- <p><span class="productname">Microsoft Windows NT</span> code is written in
- a foolproof style as it accepts even invalid input values, and which
- it usually corrects. This makes long-term debugging a pain as it hides
- sources of problems. "Checked build" releases were probably
- designed to fix this flaw by strict consistency checks but it did not reach
- its goals as such checks are usually missing in the code.</p>
-
- <p>This project has strict consistency checks across all the code to make
- the debugging phase easy enough. Failed sanity check is not always
- a bug – sometimes it just means the real W32 binary code is more
- benevolent than it could be expected according to the documentation and
- such sanity check gets removed for the next version build. In other cases
- the failed sanity checks mean the execution path for some unexpected
- arguments combination was not yet implemented by this project. I may also
- mean a bug, of course...</p>
-
- <p>Last but not least – never miss a possible sanity check as its
- later removal is in an order of magnitude cheaper than an uncaught
- invalid assumption. Failed assertion is not always a bug although it
- has to be fixed, of course.</p>
-
-
- <h2>STATUS_LOG_FILE_FULL</h2>
-
- <p>After writing approx. 1MB of data on NTFS test partition NTFS driver
- returns for any further write requests
- <span class="constant">STATUS_LOG_FILE_FULL</status> error code.
- Apparently it is caused by the fact this project is
- @{[ a_href '#synchronous','single-threaded' ]} and it ignores the spawn
- of parallel journalling thread during <span class="fname">ntfs.sys</span>
- initialization.</p>
-
- <p>Fortunately <span class="fname">ntfs.sys</span> will clear its
- journalling log file during filesystem unmount. This project will therefore
- remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</status>
- is detected to workaround missing journalling thread.</p>
-
- <p>Similiar behaviour can be seen during write of compressed files —
- the file gets written uncompressed and its compression will proceed only
- during the final filesystem unmount.</p>
-
- <p>For these reasons it was mandatory to support
- @{[ a_href '#parent_connector','transparent volume remounting' ]}.</p>
-
-
- <a name="parent_connector"><h2><span class="constant">ParentConnector</span> volume remounter</h2></a>
-
- <p>The sandbox master component of this project has control of restarting
- its sandbox slaves containing the W32 filesystem. Target goal of
- <span class="constant">ParentConnector</span> component is to transparently
- provide persistent view of files and directories over the sandboxed slaves
- being restarted.</p>
-
- <p>In the case of read-only operations it would be simple as we could only
- save our state of currently opened filesystem objects with their read
- file/directory offset. Write operations can be handled as the read-only
- ones as long as all the operations are successful. In the case of W32
- filesystem crash we loose all the past write operations. If we would redo
- all the write operations we could very easily invoke the same crash.
- Therefore we write:</p>
-
- <blockquote class="command">
- <p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
- </blockquote>
-
- <p>message to syslog and refuse any further operations with this
- object.</p>
-
- @{[ doc_img 'dia/parent-connector','Parent Connector' ]}
-
- <p><span class="constant">HANDLE</span> represents W32 object open in
- existing W32 filesystem.<span class="constant">HANDLE</span> is created
- on-demand according to the saved state of the object (such as its
- pathname). Even the whole <span class="constant">VFS</span> sandbox slave
- is spawn on-demand if some object operation requests it.</p>
-
- <p>W32 filesystem crash can obviously occur at any moment - it generates
- @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
- @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
- <span class="constant">abort</span>. Successful filesystem unmount
- (even as the part of remount operation) must be first preceded by
- <span class="constant">detach</span> signal to close all existing
- W32 <span class="constant">HANDLE</span>s. After their close the filesystem
- gets the unmount requests. Only in the case all the close operations
- succeeded including the final filesystem unmount the signal
- <span class="constant">cease</span> can be activated to notify all the
- dirty (written) objects they are now clean. During this
- <span class="constant">cease</span> signal the project will also
- @{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
- underlying media.</p>
-
- <p>Objects never written remain in <span class="constant">clean</span>
- state and they can be transparently reopened even if W32 filesystem crash
- occurs.</p>
-
-
-<h1>TODO: Fsck of NTFS</h1>
-
- <p>Currently this project does not support checking of data structures
- of NTFS volume as being provided by <span class="command">chkdsk.exe</span>
- in W32 environment and <span class="command">fsck</span> in UNIX OS.</p>
-
- <p>W32 has its disk checking functionality split to
- <span class="fname">untfs.dll</span> W32 userland library.
- according to
- @{[ a_href 'http://www.sysinternals.com/ntw2k/source/fmifs.shtml',
- 'Chkdskx and Formatx' ]}
- by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
- 'Mark Russinovich' ]}.
-
- <p>I assume its execution falls completely
- @{[ a_href '#existing_emulation','out of scope' ]}
- of this project as it is W32 userland.</p>
-
- <p>This possibility was not yet investigated in any way.</p>
-
-
-<h1>TODO: NTFS Support for
- <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span></h1>
-
- <p>Although there currently exists
- <span class="productname">@{[ a_href 'http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html','ntfsresize' ]}</span>
- I am not sure whether it is really reliable for all NTFS filesystems.
- <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span>
- is the only partition manager capable of safely resize the disk
- by using just the original W32 filesystem driver by full rebuild of
- filesystem metadata.
- Almost no file data blocks would be moved even on these generic filesystems
- as W32 supports <span class="constant">FSCTL_MOVE_FILE</span> request
- according to
- @{[ a_href 'http://www.sysinternals.com/ntw2k/info/defrag.shtml',
- 'Inside Windows NT Disk Defragmenting' ]}
- by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
- 'Mark Russinovich' ]}.
-
-
-<h1>Related Projects</h1>
-
- <p>The usual solution for file exchange between $freespeech operating systems
- and <span class="productname">Microsoft Windows NT</span> is to use
- <span class="productname">FAT32</span> (<span class="productname">vfat</span>
- called in $gnulinux) partition and swap the files over it. This method is not
- very comfortable as you never have access to all the files of the other
- operating system.</p>
-
- <a name="LinuxNTFScompet"><h2>$LinuxNTFS</h2></a>
-
- <p>Although this project takes a completely different approach and has
- a different architecture, the final goal is the same as for this
- project – reliable read-write <span class="productname">NTFS</span>
- filesystem support. $LinuxNTFS goes the way of reverse engineering
- filesystem data structures (and possibly
- <span class="fname">ntfs.sys</span> itself). Unfortunately after many years
- of its development it did not yet reach the state of reliable read-write
- access although its read-only part is considered trustworthy.</p>
-
- <p>Using $LinuxNTFS for read-only access to existing partition with
- <span class="productname">Microsoft Windows NT</span> installation is
- planned to be able to acquire existing <span class="fname">ntfs.sys</span>,
- <span class="fname">ntoskrnl.exe</span> and possibly
- <span class="fname">ksecdd.sys</span> (imported by
- <span class="fname">ntfs.sys</span>) files from the user's
- <span class="productname">NTFS</span> partition.</p>
-
- <h2><span class="productname">@{[ a_href 'http://www.cgsecurity.org/ntfs.html','NTPwd NTFS Driver' ]}</span></h2>
-
- <p>DOS based @{[ a_href 'http://www.gnu.org/licenses/gpl.html','GPL-2.0' ]}
- read-write NTFS driver. Filesystem structures are reverse engineered in the
- way of @{[ a_href '#LinuxNTFScompet','Linux-NTFS Project' ]}. As it is not very
- actively maintained it reaches a lower level of
- <span class="productname">NTFS</span> compatibility.</p>
-
- <h2>@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</h2>
-
- <p>The only real competition: Closed-source read/write @{[ '$299' ]} equivalent.</p>
-
- <p>Original Microsoft Windows operating system can be run inside a virtual
- machine running under GNU/Linux and share the read-write NTFS disk by using
- a network file sharing through a VMware virtual network card.</p>
-
- <p>You need @{[ '$299' ]} for this product and you need to
- give up your system security by running un@{[ a_href '#sandbox','sandbox' ]}ed
- closed-source program in your GNU/Linux.</p>
-
- <h2>@{[ a_href 'http://www.winehq.com/','Wine Project' ]}</h2>
-
- <p>No code could be shared – Wine emulates only Microsoft Windows userland.
- Filesystem drivers completely belong to Microsoft Windows kernelland.</p>
-
- <h2>@{[ a_href 'http://www.sysinternals.com/ntw2k/freeware/ntfswin98.shtml','NTFS for Windows 98' ]}</h2>
-
- <p>Closed-source read-only-crippled @{[ '$0' ]} equivalent for Microsoft Windows.</p>
-
- <p>There is a @{[ a_href 'http://www.sysinternals.com/images/screenshots/ntfs98ap.gif',
- 'diagram' ]} showing exactly the principle of Captive NTFS project.
- There is apparently disabled read/write functionality in <i>NTFS for
- Windows 98</i> as the same company also sells the following product sharing
- the same codebase:</p>
-
- <h2>@{[ a_href 'http://www.winternals.com/products/repairandrecovery/ntfsdospro.asp','NTFSDOS Professional' ]}</h2>
-
- <p>Closed-source read/write @{[ '$299' ]} equivalent for MS-DOS.</p>
-
- <p>This product is the most close equivalent to Captive NTFS but it is
- a commercial product, closed-source and it has filesystem interface only
- for MS-DOS.</p>
-
-
-<h1>Re: @{[ a_href 'http://linux-ntfs.sourceforge.net/info/ntfs.html#7.7',
- "7.7 Can't we write a wrapper for Windows' driver?" ]}</h1>
-
- <p class="re">> It sounds like a great idea, to start with, but there are numerous
- problems.</p>
-
- <p><span class="re">> The largest technical problem is joining the Windows
- system DLL to the Linux VFS. It could be done, but it wouldn't be pretty.</span><br />
- Yep. :-)</p>
-
- <p><span class="re">> It would have to run as part of the kernel which would mean
- that if it went wrong it could crash the machine. With no source, we might not
- be able to work around the problem.</span><br />
- @{[ a_href '#sandbox','Nope' ]},
- @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
- moves the filesystem implementation to UNIX userland where the Microsoft
- Windows filesystem is completely unarmed by Captive jail of chroot(2),
- setuid(2) and setrlimit(2). There only remains one narrow connection to the rest of
- system (by CORBA/ORBit). The filesystem's life environment gets kill(2)ed when
- UNIX is no longer satisfied with it. Safety similiar to
- @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware sandbox' ]}.</p>
-
- <p><span class="re">> The next major problem is compati<!--orig. text typo-->bility.
- Which version of the Windows system file would we use? Picking one would limit
- its use, making the wrapper versatile for all of them would be a programming
- nightmare.</span><br />
- Microsoft Windows NTFS filesystem driver is capable of accessing older formats
- of the filesystem. This project currently runs Microsoft Windows XP version,
- porting to Microsoft Windows 2003 Server expected. (Microsoft Windows upgrades
- NTFS disk filesystem to its own version during complete CD-ROM Microsoft
- Windows system installation – such operation is not threat this project use.)</p>
-
- <p><span class="re">> And it gets worse. The legal implications of
- distributing Windows systems files would cause problems.</span><br />
- User must be careful to obey all licensing restrictions according to his
- local country laws.<br />
- <span class="re">> Also the proprietary nature of the driver would mean that
- the other kernel coders would not investigate any problems if someone had used
- the NTFS wrapper.</span><br />
- It does not apply to this project due to the implemented
- @{[ a_href '#sandbox','filesystem separation' ]}.</p>
+ <li><a href="Details.html.pl#emulmeth_vm">Virtualmachine Running the Original W32 Subsystem</a></li>
+ <li><a href="Details.html.pl#method_ntoskrnl">"ntoskrnl.exe" Inside Virtual Address Space</a></li>
+ <li><a href="Details.html.pl#emulmeth_fs">Filesystem Driver Inside Virtual Address Space</a></li>
+ </ul></li>
+ <li><a href="Details.html.pl#apichoice">API Function Implementation Choices</a></li>
+ <li><a href="Details.html.pl#sandbox">Sandboxing of W32 filesystem</a></li>
+ <li><a href="Details.html.pl#patched">"patched" vs. "unpatched" Libraries</a></li>
+ <li><a href="Details.html.pl#mman">Memory Management</a></li>
+ <li><a href="Details.html.pl#unicode">Unicode Strings and Characters</a></li>
+ <li><a href="Details.html.pl#binfmt">Supported Binary Formats</a></li>
+ <li><a href="Details.html.pl#mounted_one">At Most One Mounted Filesystem</a></li>
+ <li><a href="Details.html.pl#synchronous">Multithreading and Multiple Processors</a></li>
+ <li><a href="Details.html.pl#paranoia">Paranoia Checks</a></li>
+ <li><a href="Details.html.pl#logfile">STATUS_LOG_FILE_FULL</a></li>
+ <li><a href="Details.html.pl#parent_connector"><span class="constant">ParentConnector</span> volume remounter</a></li>
+
+ <li><a href="APITypes.html.pl">API Function Implementation Choices</a>
+ <ul>
+ <li><a href="APITypes.html.pl#functype_pass">Direct Pass to Original "ntoskrnl.exe"</a>
+ <ul>
+ <li><a href="APITypes.html.pl#functype_pass_fromunix">Pass from UNIX Code</a></li>
+ <li><a href="APITypes.html.pl#functype_pass_fromw32">Pass from W32 Code</a></li>
+ </ul></li>
+ <li><a href="APITypes.html.pl#functype_wrap">Wrap of the Original "ntoskrnl.exe" Function</a>
+ <ul>
+ <li><a href="APITypes.html.pl#functype_wrap_fromunix">Wrapping of Call from UNIX Code</a></li>
+ <li><a href="APITypes.html.pl#functype_wrap_fromw32">Wrapping of Call from W32 Code</a></li>
+ </ul></li>
+ <li><a href="APITypes.html.pl#functype_native">Native Implementation</a>
+ <ul>
+ <li><a href="APITypes.html.pl#functype_native_fromunix">Native Implementation Called from UNIX Code</a></li>
+ <li><a href="APITypes.html.pl#functype_native_fromw32">Native Implementation of "unpatched"
+ Library Function Called from W32 Code</a></li>
+ <li><a href="APITypes.html.pl#functype_native_fromw32_patched">Native Implementation of "patched"
+ Library Function Called from W32 Code</a></li>
+ <li><a href="APITypes.html.pl#functype_native_reactos">Native Implementation
+ - <span class="productname">ReactOS</span></a></li>
+ <li><a href="APITypes.html.pl#functype_native_wine">Native Implementation –
+ <span class="productname">Wine</span></a></li>
+ <li><a href="APITypes.html.pl#functype_native_libcaptive">Native Implementation – Project Specific</a></li>
+ </ul></li>
+ <li><a href="APITypes.html.pl#functype_undef">Undefined Function</a></li>
+ </ul></li>
+
+ <li><a href="CallType.html.pl">API Function Calling Conventions</a>
+ <ul>
+ <li><a href="CallType.html.pl#calltype_cdecl">W32 Calling Convention "cdecl"</a></li>
+ <li><a href="CallType.html.pl#calltype_stdcall">W32 Calling Convention "stdcall"</a></li>
+ <li><a href="CallType.html.pl#calltype_fastcall">W32 Calling Convention "fastcall"</a></li>
+ </ul></li>
+ </ul></li>
+
+<li><a href="TODO.html.pl#todo_fsck">TODO: Fsck of NTFS</a></li>
+<li><a href="TODO.html.pl#todo_surprise">TODO: NTFS Support for <span class="productname">Partition Surprise</span></a></li>
+
+<li><a href="Related.html.pl">Related Projects</a>
+ <ul>
+ <li><a href="Related.html.pl#LinuxNTFScompet">$LinuxNTFS</a></li>
+ <li><a href="Related.html.pl#NTPwd"><span class="productname">NTPwd NTFS Driver</span></a></li>
+ <li><a href="Related.html.pl#vmware">VMware Workstation</a></li>
+ <li><a href="Related.html.pl#wine">Wine Project</a></li>
+ <li><a href="Related.html.pl#ntfs98">NTFS for Windows 98</a></li>
+ <li><a href="Related.html.pl#ntfsdos">NTFSDOS Professional</a></li>
+ </ul></li>
+
+<li><a href="LinuxNTFS.html.pl">Re: 7.7 Can't we write a wrapper for Windows' driver?</a></li>
+
+</ul>
HERE
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc LinuxNTFS page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::LinuxNTFS;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: Captive vs. Linux-NTFS',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<h1>Re: @{[ a_href 'http://linux-ntfs.sourceforge.net/info/ntfs.html#7.7',
+ "7.7 Can't we write a wrapper for Windows' driver?" ]}</h1>
+
+ <p class="re">> It sounds like a great idea, to start with, but there are numerous
+ problems.</p>
+
+ <p><span class="re">> The largest technical problem is joining the Windows
+ system DLL to the Linux VFS. It could be done, but it wouldn't be pretty.</span><br />
+ Yep. :-)</p>
+
+ <p><span class="re">> It would have to run as part of the kernel which would mean
+ that if it went wrong it could crash the machine. With no source, we might not
+ be able to work around the problem.</span><br />
+ @{[ a_href '#sandbox','Nope' ]},
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
+ moves the filesystem implementation to UNIX userland where the Microsoft
+ Windows filesystem is completely unarmed by Captive jail of chroot(2),
+ setuid(2) and setrlimit(2). There only remains one narrow connection to the rest of
+ system (by CORBA/ORBit). The filesystem's life environment gets kill(2)ed when
+ UNIX is no longer satisfied with it. Safety similiar to
+ @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware sandbox' ]}.</p>
+
+ <p><span class="re">> The next major problem is compati<!--orig. text typo-->bility.
+ Which version of the Windows system file would we use? Picking one would limit
+ its use, making the wrapper versatile for all of them would be a programming
+ nightmare.</span><br />
+ Microsoft Windows NTFS filesystem driver is capable of accessing older formats
+ of the filesystem. This project currently runs Microsoft Windows XP version,
+ porting to Microsoft Windows 2003 Server expected. (Microsoft Windows upgrades
+ NTFS disk filesystem to its own version during complete CD-ROM Microsoft
+ Windows system installation – such operation is not threat this project use.)</p>
+
+ <p><span class="re">> And it gets worse. The legal implications of
+ distributing Windows systems files would cause problems.</span><br />
+ User must be careful to obey all licensing restrictions according to his
+ local country laws.<br />
+ <span class="re">> Also the proprietary nature of the driver would mean that
+ the other kernel coders would not investigate any problems if someone had used
+ the NTFS wrapper.</span><br />
+ It does not apply to this project due to the implemented
+ @{[ a_href '#sandbox','filesystem separation' ]}.</p>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc macros.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::Macros;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+use Exporter;
+our @EXPORT=qw(
+ &doc_img &productname &captive_srcfile
+ $doc_Macros_head_css $freespeech $freebeer $Wine $ReactOS $LinuxNTFS $GnomeVFS $GnomeVFSmodule $gnulinux
+ );
+our @ISA=qw(Exporter);
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+
+
+our $doc_Macros_head_css="
+.productname { font-family: cursive; }
+.fname { font-family: monospace; }
+.constant { font-family: monospace; }
+.author { font-family: cursive; }
+.stuff { font-style: italic; font-size: larger; margin-left: 20%; margin-right: 10%; }
+.function { font-family: monospace; }
+.type { font-family: monospace; }
+.command { font-family: monospace; }
+.instruction { font-style: italic; }
+";
+
+
+sub doc_img ($$)
+{
+my($img_base,$caption)=@_;
+
+ my $r="";
+ $r.='<table border="0" align="center">'."\n";
+ $r.="\t<tr><td>".img($img_base,$caption)."</td></tr>\n";
+ $r.="\t<caption>$caption</caption>\n";
+ $r.='</table>'."\n";
+ $r.=vskip "2ex";
+ return $r;
+}
+
+sub captive_srcfile ($;$)
+{
+my($filename,$text)=@_;
+
+ a_href 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/'.$filename.'?rev=HEAD',
+ ($text || $filename);
+}
+
+our $freespeech=a_href 'http://www.gnu.org/philosophy/free-sw.html','Free';
+our $freebeer=a_href 'http://www.gnu.org/philosophy/free-sw.html','free (as in beer)';
+
+sub productname
+{
+my($url,$name)=@_;
+
+ return '<span class="productname">'.a_href($url,CGI::escapeHTML($name)).'</span>';
+}
+our $Wine=productname 'http://www.winehq.com/','Wine';
+our $ReactOS=productname 'http://www.reactos.com/','ReactOS';
+our $LinuxNTFS=productname 'http://linux-ntfs.sourceforge.net/','Linux NTFS';
+our $GnomeVFS=productname 'http://developer.gnome.org/doc/API/gnome-vfs/','Gnome-VFS';
+our $GnomeVFSmodule=productname 'http://developer.gnome.org/doc/API/gnome-vfs/modules.html','Gnome-VFS-module';
+our $gnulinux='GNU/Linux';
+
+
+1;
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Related Projects page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::Related;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: Related Projects',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<h1>Related Projects</h1>
+
+ <p>The usual solution for file exchange between $freespeech operating systems
+ and <span class="productname">Microsoft Windows NT</span> is to use
+ <span class="productname">FAT32</span> (<span class="productname">vfat</span>
+ called in $gnulinux) partition and swap the files over it. This method is not
+ very comfortable as you never have access to all the files of the other
+ operating system.</p>
+
+ <a name="LinuxNTFScompet"><h2>$LinuxNTFS</h2></a>
+
+ <p>Although this project takes a completely different approach and has
+ a different architecture, the final goal is the same as for this
+ project – reliable read-write <span class="productname">NTFS</span>
+ filesystem support. $LinuxNTFS goes the way of reverse engineering
+ filesystem data structures (and possibly
+ <span class="fname">ntfs.sys</span> itself). Unfortunately after many years
+ of its development it did not yet reach the state of reliable read-write
+ access although its read-only part is considered trustworthy.</p>
+
+ <p>Using $LinuxNTFS for read-only access to existing partition with
+ <span class="productname">Microsoft Windows NT</span> installation is
+ planned to be able to acquire existing <span class="fname">ntfs.sys</span>,
+ <span class="fname">ntoskrnl.exe</span> and possibly
+ <span class="fname">ksecdd.sys</span> (imported by
+ <span class="fname">ntfs.sys</span>) files from the user's
+ <span class="productname">NTFS</span> partition.</p>
+
+ <a name="NTPwd"><h2><span class="productname">@{[ a_href 'http://www.cgsecurity.org/ntfs.html',
+ 'NTPwd NTFS Driver' ]}</span></h2></a>
+
+ <p>DOS based @{[ a_href 'http://www.gnu.org/licenses/gpl.html','GPL-2.0' ]}
+ read-write NTFS driver. Filesystem structures are reverse engineered in the
+ way of @{[ a_href '#LinuxNTFScompet','Linux-NTFS Project' ]}. As it is not very
+ actively maintained it reaches a lower level of
+ <span class="productname">NTFS</span> compatibility.</p>
+
+ <a name="vmware"><h2>@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</h2></a>
+
+ <p>The only real competition: Closed-source read/write @{[ '$299' ]} equivalent.</p>
+
+ <p>Original Microsoft Windows operating system can be run inside a virtual
+ machine running under GNU/Linux and share the read-write NTFS disk by using
+ a network file sharing through a VMware virtual network card.</p>
+
+ <p>You need @{[ '$299' ]} for this product and you need to
+ give up your system security by running un@{[ a_href '#sandbox','sandbox' ]}ed
+ closed-source program in your GNU/Linux.</p>
+
+ <a name="wine"><h2>@{[ a_href 'http://www.winehq.com/','Wine Project' ]}</h2></a>
+
+ <p>No code could be shared – Wine emulates only Microsoft Windows userland.
+ Filesystem drivers completely belong to Microsoft Windows kernelland.</p>
+
+ <a name="ntfs98"<h2>@{[ a_href 'http://www.sysinternals.com/ntw2k/freeware/ntfswin98.shtml','NTFS for Windows 98' ]}</h2></a>
+
+ <p>Closed-source read-only-crippled @{[ '$0' ]} equivalent for Microsoft Windows.</p>
+
+ <p>There is a @{[ a_href 'http://www.sysinternals.com/images/screenshots/ntfs98ap.gif',
+ 'diagram' ]} showing exactly the principle of Captive NTFS project.
+ There is apparently disabled read/write functionality in <i>NTFS for
+ Windows 98</i> as the same company also sells the following product sharing
+ the same codebase:</p>
+
+ <a name="ntfsdos"><h2>@{[ a_href 'http://www.winternals.com/products/repairandrecovery/ntfsdospro.asp',
+ 'NTFSDOS Professional' ]}</h2></a>
+
+ <p>Closed-source read/write @{[ '$299' ]} equivalent for MS-DOS.</p>
+
+ <p>This product is the most close equivalent to Captive NTFS but it is
+ a commercial product, closed-source and it has filesystem interface only
+ for MS-DOS.</p>
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc Reverse Engineering page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::Reverse;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: Reverse Engineering',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<a name="reverse"><h1>Reverse Engineering</h1></a>
+
+ <p>This project has no intentions to reverse engineer and document the
+ filesystem data structures themselves since they are being encapsulated by
+ the filesystem driver. For these reasons the resources available in
+ projects such as $LinuxNTFS get out of any possible use. This project goal
+ is to provide fully compatible API interface to the rest of the W32 system
+ to persuade the filesystem driver it is running in the native
+ <span class="productname">Microsoft Windows XP</span> environment.</p>
+
+ <p>All the W32 filesystem drivers are running in the W32 kernel address
+ space and this area of W32 API is not much documented by
+ <span class="productname">Microsoft</span>. Some API functions are not
+ documented at all and the others are documented insufficiently for a their
+ possibly needed reimplementation from scratch. Documentation being
+ consulted primarily consists of
+ <span class="productname">@{[ a_href 'http://msdn.microsoft.com/library/default.asp?url=/library/en-us/kmarch/hh/kmarch/kmhdr_6enb.asp','MSDN (Microsoft Developer Network) Kernel-Mode Driver Architecture: Windows DDK' ]}</span>
+ documentation and also various other 3rd party documentation resources such as
+ <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
+ 'The NT Cache Manager Description' ]}</span>,
+ <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
+ 'Learn About NT'."'".'s File-system Cache' ]}</span>,
+ <span class="productname">@{[ a_href 'http://www.ntfsd.org/archive/',
+ 'NT File System Developers mailing list archives' ]}</span>
+ including various
+ @{[ a_href 'http://www.google.com/search?q=site%3Amicrosoft.com','fulltext searches' ]}
+ through Internet from case to case.</p>
+
+ <p>Sometimes no sufficient documentation was found and some code behaviour
+ had to be reverse engineered directly from the binaries of
+ <span class="fname">ntoskrnl.exe</span>,
+ <span class="fname">cdfs.sys</span>,
+ <span class="fname">fastfat.sys</span>
+ and primarily
+ <span class="fname">ntfs.sys</span>.
+ Up to now the code was disassembled by
+ <span class="productname">@{[ a_href 'http://www.simtel.net/pub/pd/29498.html','IDA Freeware' ]}</span>
+ and by
+ <span class="productname">dumpbin.exe</span> of
+ <span class="productname">Microsoft Visual Studio</span>.
+ <span class="productname">dumpbin.exe</span> is fortunately able to
+ interpret debug symbols from W32 <span class="fname">.PDB</span>
+ (Program DataBase) debug information files.</p>
+
+ <a name="dumpbin"><h2><span class="productname">dumpbin.exe</span></h2></a>
+
+ <p>You should use the following options for
+ <span class="productname">dumpbin.exe</span>:</p>
+
+ <blockquote class="command">
+ <p>dumpbin.exe /all /rawdata:none /disasm /pdbpath:verbose FILENAME.SYS</p>
+ </blockquote>
+
+ <p>You should see the following line in the output:</p>
+
+ <blockquote class="command">
+ <p>PDB file found at '.\\FILENAME.pdb'</p>
+ </blockquote>
+
+ <a name="WinDbg"><h2><span class="productname">WinDbg</span> Windows NT kernel debugging</h2></a>
+
+ <p><span class="productname">WinDbg</span> is downloadable from:
+ @{[ a_href 'http://www.microsoft.com/whdc/ddk/debugging/installx86.mspx' ]}</p>
+
+ <p>This is (the only?) tool able to debug filesystem drivers incl.
+ <span class="fname">ntfs.sys</span>. You will need two computers running
+ <span class="productname">Microsoft Windows</span> — one computer will run
+ <span class="productname">WinDbg</span> while the other one will be
+ frozen in remote Windows NT kernel debug mode. It does not matter which
+ <span class="productname">Microsoft Windows</span> version will be run
+ on the <span class="productname">WinDbg</span> side.</p>
+
+ <p>The most easy way to setup two computers is to use commercial
+ <span class="productname">@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</span>
+ where you can run two virtual machines simultaneously on single PC
+ hardware and you can connect them by a virtual serial port provided by
+ <span class="productname">VMware</span>.</p>
+
+ <a name="WinDbg_WinDbg"><h3><span class="productname">WinDbg</span> side setup</h3></a>
+
+ @{[ doc_img 'ntdebug-vmware-windbg',
+ '<span class="productname">VMware</span> virtual serial port'
+ .' of <span class="productname">WinDbg</span> side' ]}
+
+ <p>You should setup <span class="productname">WinDbg</span> according
+ to:</p>
+
+ @{[ doc_img 'ntdebug-windbg-port','Port settings of <span class="productname">WinDbg</span>' ]}
+ @{[ doc_img 'ntdebug-windbg-sym','Symbols files location of <span class="productname">WinDbg</span>' ]}
+
+ <span class="constant">Symbols</span> should point to the directory where
+ reside files extracted from the symbol archive for your version of
+ <span class="productname">Microsoft Windows</span>. In the case of the
+ recommended <span class="productname">Microsoft Windows XP Service Pack 1 Checked Build</span>
+ you should use:
+ @{[ a_href 'http://msdl.microsoft.com/download/symbols/packages/windowsxp/xpsp1sym_x86_chk.exe' ]}</p>
+
+ <blockquote class="command">
+ <p># Rename xpsp1sym_x86_chk.exe contents .pdb files for WinDbg<br />
+ @{[ CGI::escapeHTML(q{for i in *.pdb*;do ext="`echo $i|sed 's/^.*\.pdb\.\(.*\)$/\1/'`";if [ "$i" = "$ext" ];then echo "BAD:$i";break;fi;base="`echo $i|sed 's/\(\.pdb\)\..*$/\1/'`";echo "md $ext";echo "move /-y $i $ext\\$base";done|sort -u|sed 's/$/'`echo -ne '\r'`'/g' >/tmp/rename.bat}) ]}</p>
+ </blockquote>
+
+ <p>The resulting <span class="command">rename.bat</span> for
+ <span class="command">xpsp1sym_x86_chk.exe</span> can be found at:
+ @{[ a_href 'xpsp1sym_x86_chk-rename.bat.zip' ]}</p>
+
+ <p>The resulting directory should contain at least
+ <span class="command">sys\\ntfs.pdb</span>
+ and
+ <span class="command">exe\\ntoskrnl.pdb</span>.</p>
+
+ <p>Your successfuly connected target (after the steps described
+ below) should look like:</p>
+
+ @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}
+
+ <a name="WinDbg_kern"><h3>Setup of the side being kernel-debugged</h3></a>
+
+ @{[ doc_img 'ntdebug-vmware-xpdebug',
+ '<span class="productname">VMware</span> virtual serial port'
+ .' of the side being kernel-debugged' ]}
+
+ <p>You must use the following options in your
+ <span class="command">c:\\boot.init</span> command-line:</p>
+
+ <blockquote class="command">
+ <p>/debug /debugport=COM1 /baudrate=115200</p>
+ </blockquote>
+
+ <p>After booting this <span class="command">boot.ini</span>-entry
+ should freeze at this point
+ (if no <span class="productname">WinDbg</span> is waiting in the other
+ virtual machine):</p>
+
+ @{[ doc_img 'ntdebug-wait','Side being kernel-debugged waiting for <span class="productname">WinDbg</span>' ]}
+
+
+HERE
+
+
+My::Web->footer();
--- /dev/null
+#! /usr/bin/perl
+#
+# $Id$
+# Captive project doc TODO page Perl template.
+# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; exactly version 2 of June 1991 is required
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+package project::captive::doc::TODO;
+require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
+our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
+our $CVS_ID=q$Id$;
+use strict;
+use warnings;
+
+BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
+use My::Web;
+require "CGI";
+use project::captive::doc::Macros;
+
+
+My::Web->init(
+ "__PACKAGE__"=>__PACKAGE__,
+ "title"=>'Captive NTFS Developer Documentation: TODO',
+ "head_css"=>$doc_Macros_head_css,
+ );
+My::Web->heading();
+
+
+print <<"HERE";
+
+
+<a name="todo_fsck"><h1>TODO: Fsck of NTFS</h1></a>
+
+ <p>Currently this project does not support checking of data structures
+ of NTFS volume as being provided by <span class="command">chkdsk.exe</span>
+ in W32 environment and <span class="command">fsck</span> in UNIX OS.</p>
+
+ <p>W32 has its disk checking functionality split to
+ <span class="fname">untfs.dll</span> W32 userland library.
+ according to
+ @{[ a_href 'http://www.sysinternals.com/ntw2k/source/fmifs.shtml',
+ 'Chkdskx and Formatx' ]}
+ by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
+ 'Mark Russinovich' ]}.
+
+ <p>I assume its execution falls completely
+ @{[ a_href '#existing_emulation','out of scope' ]}
+ of this project as it is W32 userland.</p>
+
+ <p>This possibility was not yet investigated in any way.</p>
+
+
+<a name="todo_surprise"><h1>TODO: NTFS Support for
+ <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span></h1></a>
+
+ <p>Although there currently exists
+ <span class="productname">@{[ a_href 'http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html','ntfsresize' ]}</span>
+ I am not sure whether it is really reliable for all NTFS filesystems.
+ <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span>
+ is the only partition manager capable of safely resize the disk
+ by using just the original W32 filesystem driver by full rebuild of
+ filesystem metadata.
+ Almost no file data blocks would be moved even on these generic filesystems
+ as W32 supports <span class="constant">FSCTL_MOVE_FILE</span> request
+ according to
+ @{[ a_href 'http://www.sysinternals.com/ntw2k/info/defrag.shtml',
+ 'Inside Windows NT Disk Defragmenting' ]}
+ by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
+ 'Mark Russinovich' ]}.
+
+
+HERE
+
+
+My::Web->footer();
git://git.jankratochvil.net / www.jankratochvil.net.git / commitdiff
