+++ /dev/null
-#! /usr/bin/perl
-#
-# $Id$
-# Captive project doc Details page Perl template.
-# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; exactly version 2 of June 1991 is required
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-
-package project::captive::doc::Details;
-require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
-our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
-our $CVS_ID=q$Id$;
-use strict;
-use warnings;
-
-BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
-use My::Web;
-require CGI;
-BEGIN { Wuse 'project::captive::doc::Macros'; }
-
-
-project::captive::doc::Macros->init(
- "__PACKAGE__"=>__PACKAGE__,
- "title"=>'Captive NTFS Developer Documentation: Implementation Details',
- "rel_prev"=>'CacheManager.html.pl',
- "rel_next"=>'APITypes.html.pl',
- );
-
-
-print <<"HERE";
-
-
-<h1>Implementation Details</h1>
-
- <a name="emulmeth"><h2>Choice of the Emulation Methods</h2></a>
-
- <p>The intent of the project was to get reliable read-write access to
- <span class="productname">NTFS</span> partition. There are several possible
- ways to achieve that:</p>
-
- <a name="emulmeth_vm"><h3>Virtualmachine Running the Original W32 Subsystem</h3></a>
-
- <p>Creating virtual-hardware PC and running the original W32 binaries
- including their boot-loader etc. Disk device access would be passed as
- virtual IDE disk (=hard disk drive). File access API would be implemented
- either by special escaping by some trapped instruction out of the
- virtualmachine while using W32 file access API or using the standard W32
- SMB (Server Message Block) network access through some virtual network
- card. The latter network access solution is almost the currently available
- possibility of running full-blown disk-sharing real
- <span class="productname">Microsoft Windows NT</span> inside virtual
- machine emulator such as <span class="productname">VMware</span>.</p>
-
- <p>pros: Full compatibility due to fully native codebase.</p>
-
- <p>cons: Hard to debug, missing documentation of NT booting internals,
- possible problems by different PC virtual-hardware than expected by NT,
- requirement of fully installed
- <span class="productname">Microsoft Windows NT</span> product.</p>
-
- <a name="method_ntoskrnl"><h3>"ntoskrnl.exe" Inside Virtual Address Space</h3></a>
-
- <p>This solution was chosen by the project. Binary filesystem driver and
- also <span class="fname">ntoskrnl.exe</span> binary file are required.
- Unfortunately <span class="fname">ntoskrnl.exe</span> expects a native
- PC virtual-hardware missing during regular UNIX user space process
- emulation, therefore such instructions must be trapped and emulated/ignored
- from case to case.</p>
-
- <p>Also the <a name="init_ntoskrnl">initialization code of <span
- class="fname">ntoskrnl.exe</span></a> is not executed by this project since
- it expects to get full PC hardware access privileges and thus some
- datastructures do not get initialized by it (need to be trapped later at
- runtime stage). Some of the missing initializations are solved by
- @{[ a_href 'APITypes.html.pl#functype_wrap','API functions wrapping' ]}.
-
- <p>pros: Lightweight, easier to debug.</p>
-
- <p>cons: Possible incompatible emulation of
- <span class="fname">ntoskrnl.exe</span> parts, missing documentation needed
- for the implementation.</p>
-
- <a name="emulmeth_fs"><h3>Filesystem Driver Inside Virtual Address Space</h3></a>
-
- <p>Unlike @{[ a_href 'Details.html.pl#method_ntoskrnl','previous method' ]} here we do not use
- even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
- W32 is <a name="native_ntoskrnl">emulated from the project source
- files</a>. <span class="fname">cdfs.sys</span> driver was successfuly ran
- in this manner in the former versions of this project but the possibility
- to run without <span class="fname">ntoskrnl.exe</span> was dropped since it
- had no licensing gains (you need the original
- <span class="productname">Microsoft Windows NT</span> files at least for
- the filesystem driver itself) and the emulation of undocumented parts
- reusable from <span class="fname">ntoskrnl.exe</span> binary was
- a pain.</p>
-
- <p>pros: Lightweight, easier to debug.</p>
-
- <p>cons: Possible incompatible emulation of the whole
- <span class="fname">ntoskrnl.exe</span>, its missing documentation.</p>
-
-
- <a name="apichoice"><h2>API Function Implementation Choices</h2></a>
-
- <p>During the initial point of the project development all the API
- functions were defined as unimplemented, of course. Any call of such
- unimplemented function is fatal and results in program termination. When we
- need to implement any required API function we have multiple choices to do
- so:
- @{[ a_href 'APITypes.html.pl#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
- @{[ a_href 'APITypes.html.pl#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
- @{[ a_href 'APITypes.html.pl#functype_native_reactos','Native implementation – $ReactOS' ]},
- @{[ a_href 'APITypes.html.pl#functype_native_wine','Native implementation – $Wine' ]}
- or
- @{[ a_href 'APITypes.html.pl#functype_native_libcaptive','Native implementation – project specific' ]}.
- <!-- a_href 'APITypes.html.pl#functype_undef','Undefined function' -->
-
-
- <a name="sandbox"><h2>Sandboxing of W32 Filesystem</h2></a>
-
- <p>The emulated W32 environment running the original W32 filesystem driver
- is separated from the rest of UNIX OS. It achieves the following goals:</p>
-
- <ul>
- <li><b>Restartable</b>: W32 driver can be restartde in clean state if it crashed</li>
- <li><b>Secure</b>: Malicious W32 code cannot affect the security of UNIX OS</li>
- <li><b>Stable</b>: Buggy W32 cannot crash any part of UNIX OS</li>
- </ul>
-
- <p>Sandboxing is provided with the following attributes:</p>
-
- <ul>
- <li>standalone UNIX process with separate memory space</li>
- <li>chroot(2) in empty directory to prevent any UNIX OS filesystem access</li>
- <li>setuid(2) to own user/group to prevent interaction with UNIX processes</li>
- <li>setrlimit(2) to limit system resources available for W32 environment</li>
- <li>the only connection with the UNIX OS by CORBA/ORBit RPC</li>
- </ul>
-
- @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}
-
- <p>This security is almost the same as provided by
- emulated virtual machines such as
- @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware' ]}.</p>
-
- @{[ doc_img 'dia/inheritance','Sandboxing Scheme' ]}
-
- <p>Project can be also used in non-sandboxed mode by
- <span class="command">--no-sandbox</span> option as it is easier to debug
- without CORBA/ORBit RPC. In this case the
- <span class="type">DirectorySlave</span>/<span class="type">FileSlave</span>
- options are used directly instead of their
- <span class="type">DirectoryParent</span>/<span class="type">FileParent</span>
- peers.</p>
-
-
- <a name="patched"><h2>"patched" vs. "unpatched" Libraries</h2></a>
-
- <p>Library is called <span class="constant">patched</span> if we require
- loading its original binary code file. Project needs to patch it to be able
- to trap all the function entry points. The only currently
- <span class="constant">patched</span> library of this project is
- <span class="fname">ntoskrnl.exe</span>.</p>
-
- <p>Library is called <span class="constant">unpatched</span> if no original
- binary code is needed since all of its functions are completely emulated by
- @{[ a_href 'APITypes.html.pl#functype_native','the native implementations' ]} of this project.
- The typical <span class="constant">unpatched</span> representative is
- <span class="fname">hal.dll</span> as it specializes on the hardware
- dependent code and therefore it must be completely replaced by this project
- running in the $gnulinux operating system environment. Early versions of
- this project had also full <span class="constant">unpatched</span>
- <a href="#native_ntoskrnl">native implementation of
- <span class="fname">ntoskrnl.exe</span></a> but it no longer applies.</p>
-
- <a name="mman"><h2>Memory Management</h2></a>
-
- <p>Original <span class="productname">Microsoft Windows NT</span>
- architecture uses two address space areas – user space and kernel space.
- User space is mapped in the range <span class="constant">0x00000000</span>
- to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
- range <span class="constant">0x80000000</span>
- (<span class="constant">KERNEL_BASE</span> in $ReactOS sources) to
- <span class="constant">0xFFFFFFFF</span>. All these virtual memory ranges
- represent addresses after their MMU (Memory Management Unit) mapping, of
- course. More discussion can be found in the
- <a href="http://www.microsoft.com/hwdev/platform/server/PAE/PAEmem.asp">description
- by <span class="productname">Microsoft</span></a>.</p>
-
- <p>This project runs in the virtual address space used both for the UNIX
- user space process part and for the W32 kernel space. Therefore this
- project defines that W32 kernel runs in the whole range
- <span class="constant">0x00000000</span> to
- <span class="constant">0xFFFFFFFF</span> since there are no special mapping
- assumptions about the UNIX user space process mapping. No W32 user space
- exists in this project. Such approach also nullifies any special memory
- moving operations between W32 kernel space and W32 user space memory areas
- (such as <span class="function">MmSafeCopyToUser()</span>).</p>
-
- <a name="unicode"><h2>Unicode Strings and Characters</h2></a>
-
- <p>W32 platform uses 16-bit type <span class="type">wchar_t</span> while $gnulinux uses a
- 32-bit one. This can be problem during GCC (GNU C Compiler)
- compilation of combination of native UNIX C sources (assuming 32-bit
- GCC with 32-bit <span class="type">wchar_t</span>) and
- $ReactOS C sources (assuming W32 compiler with 16-bit
- <span class="type">wchar_t</span>) for literal wide strings
- (C source file systax: <span class="command">L"wstring"</span>).
- Possibilities to solve this issue list:</p>
-
- <ul>
- <li>
- <p>Using <span class="constant">-fshort-wchar</span> GCC option and
- strictly differentiate between compilation of
- <span class="productname">ReactOS</span> code and UNIX code.</p>
-
- <p>pros: No source modifications needed, no runtime performance hit.</p>
-
- <p>cons: No type checking if some part of code has bad compilation
- flags, complicated way to completely split
- <span class="productname">ReactOS</span> and UNIX code.</p>
- </li>
- <li>
- <p>Wrap all <span class="productname">ReactOS</span> literal constants
- by some conversions function call (implemented as macro
- <span class="function">REACTOS_UCS2()</span> by this project).</p>
-
- <p>pros: Any forgotten/mistaken conversions are type-checked and warned
- during the compilation by GCC.</p>
-
- <p>cons: All compiled <span class="productname">ReactOS</span> sources
- files containing literal wide strings have to be wrapped/modified,
- performance hit by runtime string conversions.</p>
-
- <p>This solution was chosen to get the internal sanity checking
- benefit.</p>
- </li>
- </ul>
-
- <a name="binfmt"><h2>Supported Binary Formats</h2></a>
-
- <p>The native W32 binary format is identified as
- <span class="constant">PE-32</span> (Portable Executable 32-bit), such
- files have all the usual extensions such as
- <span class="fname">.sys</span>, <span class="fname">.exe</span>,
- <span class="fname">.dll</span> etc. <span class="constant">PE-32</span>
- loading support was already implemented by $ReactOS, its memory mapping
- specifics just had to be ported to $gnulinux environment by this project.
- This loading support does not (yet) cover importing of debug symbols from
- W32 <span class="fname">.PDB</span> (Program DataBase) files in $gnulinux
- ABI (Application Binary Interface) compatible way.</p>
-
- <p>This project also supports transparent loading of UNIX
- <span class="fname">.so</span> (Shared Object file) binary format. If you
- have W32 source files for some W32 library you can try to compile it by GCC
- to get the shared library with $gnulinux ABI compatible debug information
- (GCC option <span class="constant">-ggdb3</span> recommended). Beware of
- possible compilation problems as <span class="productname">Microsoft</span>
- C code expects <span class="constant">exception</span> handling to be
- supported by the compiler (definitely not the case of the plain C compiler
- of GCC) — all the exception catching code should be discarded as any
- @{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
- such driver is running in the scope of this project. You can use the
- following script of this project to compile W32 filesystem source files as
- UNIX <span class="fname">.so</span>:
- @{[ captive_srcfile 'src/w32-mod/ext2fsd.so-build.sh' ]}</p>
-
- <p>Be aware of some differences if you use
- <span class="constant">PE-32</span> binary format file vs.
- <span class="fname">.so</span> format file.
- <span class="constant">PE-32</span> use the appropriate W32 specific
- @{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
- <span class="fname">.so</span> must be completely compiled in the standard
- UNIX @{[ a_href 'CallType.html.pl#calltype_cdecl','cdecl call type semantics' ]}.
- @{[ a_href 'APITypes.html.pl#functype_native','Native function implementations' ]} do not need
- to be explicitely exported by <span class="fname">captivesym</span> as they
- are resolved automatically by the UNIX dynamic system linker. It may be
- surprising you will have to fix all such missing symbol exports if you
- advance during the development from the debugging
- <span class="fname">.so</span> file for the production version of the
- original <span class="constant">PE-32</span> binary file.</p>
-
-
- <a name="mounted_one"><h2>At Most One Mounted Filesystem</h2></a>
-
- <p>The project technically supports only one (exactly one...) mounted
- filesystem device and only one filesystem driver. There is nothing
- complicated to support multiple disks and multiple loaded filesystem
- modules but as they would share the address space it would only bring
- a possible complications during bug reports and the bug solving
- itself. It was considered as a more sane way to support multiple W32
- mounted disks by completely separately running project instances in
- a different UNIX processes communicating from their sandboxes via
- @{[ a_href 'Details.html.pl#sandbox','CORBA sandbox interface' ]}. This sandboxing
- feature is not yet deployed although its code is already prepared.</p>
-
- <p>The project also does not support any state cleanup to be able to load
- filesystem <span class="constant">A</span>,
- cleanup <span class="constant">A</span> and load a different
- filesystem <span class="constant">B</span> in the same process address
- space. It complies with the preventions of the possible debugging
- complications as noted above. Despite this you still must call the function
- <span class="function">captive_shutdown()</span> to flush all the pending
- filesystem buffers to the disk. After calling
- <span class="function">captive_shutdown()</span> the process address space is
- no longer usable for any further project operations and the process is
- expected to be terminated in the manner compatible with its driving
- @{[ a_href 'Details.html.pl#sandbox','CORBA sandbox interface' ]} control master.</p>
-
- <p>Each sandbox executing the untrusted W32 binary filesystem driver code
- is connected through its
- @{[ a_href 'Details.html.pl#sandbox','CORBA sandbox interface' ]} at the point of upper
- layer <span class="constant">libcaptive</span>-specific filesystem API, at
- the point of the bottom layer of <span class="type">GIOChannel</span>
- device access and also for transfers of GLib logging
- messages/warnings/errors out of the sandbox to the user.</p>
-
-
- <a name="synchronous"><h2>Multithreading and Multiple Processors</h2></a>
-
- <p>W32 platform stands on its thorough architecture parallelism. It
- must lock all its objects to maintain coherence in presence of
- multithreading and multiple processors. Since the author of this project
- considers any parallel execution a serious obstacle for debugging the whole
- project architecture was designed to prevent any undeterministic behaviour.
- Therefore this projects always emulates uniprocessor
- <span class="productname">Microsoft Windows NT</span> kernel
- (<span class="constant">KeNumberProcessors</span> symbol is always 1),
- everything runs in the single initial thread/process and all the filesystem
- operations are performed as synchronous
- ("synchronous" by flags
- <span class="constant">FILE_SYNCHRONOUS_IO_ALERT</span>,
- <span class="constant">FO_SYNCHRONOUS_IO</span>,
- <span class="constant">IRP_SYNCHRONOUS_API</span>,
- <span class="constant">IRP_SYNCHRONOUS_PAGING_IO</span>,
- forced <span class="constant">TRUE</span> result of
- <span class="function">IoIsOperationSynchronous()</span>
- etc.).
- For several cases needed only by <span class="fname">ntfs.sys</span> there
- had to be supported asynchronous access
- (<span class="constant">STATUS_PENDING</span> return code) – parallel
- execution is emulated by GLib
- <span class="function">g_idle_add_full()</span> with
- <span class="function">g_main_context_iteration()</span> called during
- <span class="function">KeWaitForSingleObject()</span>.</p>
- Since there is a possibility a real W32 parallel threading would
- be yet needed in the future all the code that would be hit by W32
- multithreading capability is marked by
- <span class="constant">TODO:thread</span> comment.</p>
-
- <p>Multiple processors (SMP) support will never need to be implemented
- since uniprocessor W32 kernels apparently run the filesystem driver modules
- fine. As this project implements only the uniprocessor W32 kernel all the
- processor locking functions and structures such as
- <span class="constant">KSPIN_LOCK</span> etc. can be safely implemented as
- no-operations.</p>
-
- <p>Asynchronous callbacks registered for
- <span class="constant">IO_WORKITEM</span>s are passed as GLib idle
- functions by <span class="function">g_idle_add_full()</span>. Although they
- will probably never be executed during non-interactive project's batch
- executions it is the responsibility of W32 driver implementation to
- complete all the pending tasks before its W32 shutdown. Such W32 shutdown
- is done during cleanup of the project's execution by
- <span class="function">captive_shutdown()</span>.</p>
-
- <a name="paranoia"><h2>Paranoia Checks</h2></a>
-
- <p>A general approach of software projects development is to implement
- many internal sanity checks during the development stage but to produce the
- most optimized final release product without those debugging checks.</p>
-
- <p>Facilities for these practices can be seen in the standard
- C include files for example as function
- <span class="function">assert()</span> which gets disabled by the
- <span class="constant">NDEBUG</span> symbol used during the final optimized
- executable compilation. This project uses Gnome GLib messaging subsystem
- offering sanity checks discarded by symbols
- <span class="constant">G_DISABLE_ASSERT</span> and
- <span class="constant">G_DISABLE_CHECKS</span>.
- <span class="productname">Microsoft</span> also produces two versions of
- its products – regular customers use the "free build" (also
- called "retail") while the programmers should develop their code
- on the "checked build" product releases.</p>
-
- <p>As this project will always run unknown binary code of proprietary W32
- filesystem drivers, the code can never be trusted. Such code even runs in
- the same unprotected address space as its controlling UNIX code. Since
- there is not enough documentation for the W32 components of the system and
- also such documentation is usually misleading it can never be considered as
- 100% emulation. Even in the final releases all the sanity checks
- implemented in this project should remain active as all the project's code
- always interacts with unknown and untrusted W32 binaries.</p>
-
- <p><span class="productname">Microsoft Windows NT</span> code is written in
- a foolproof style as it accepts even invalid input values, and which
- it usually corrects. This makes long-term debugging a pain as it hides
- sources of problems. "Checked build" releases were probably
- designed to fix this flaw by strict consistency checks but it did not reach
- its goals as such checks are usually missing in the code.</p>
-
- <p>This project has strict consistency checks across all the code to make
- the debugging phase easy enough. Failed sanity check is not always
- a bug – sometimes it just means the real W32 binary code is more
- benevolent than it could be expected according to the documentation and
- such sanity check gets removed for the next version build. In other cases
- the failed sanity checks mean the execution path for some unexpected
- arguments combination was not yet implemented by this project. I may also
- mean a bug, of course...</p>
-
- <p>Last but not least – never miss a possible sanity check as its
- later removal is in an order of magnitude cheaper than an uncaught
- invalid assumption. Failed assertion is not always a bug although it
- has to be fixed, of course.</p>
-
-
- <a name="logfile"><h2>STATUS_LOG_FILE_FULL</h2></a>
-
- <p>After writing approx. 1MB of data on NTFS test partition NTFS driver
- returns for any further write requests
- <span class="constant">STATUS_LOG_FILE_FULL</status> error code.
- Apparently it is caused by the fact this project is
- @{[ a_href 'Details.html.pl#synchronous','single-threaded' ]} and it ignores the spawn
- of parallel journalling thread during <span class="fname">ntfs.sys</span>
- initialization.</p>
-
- <p>Fortunately <span class="fname">ntfs.sys</span> will clear its
- journalling log file during filesystem unmount. This project will therefore
- remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</status>
- is detected to workaround missing journalling thread.</p>
-
- <p>Similiar behaviour can be seen during write of compressed files —
- the file gets written uncompressed and its compression will proceed only
- during the final filesystem unmount.</p>
-
- <p>For these reasons it was mandatory to support
- @{[ a_href 'Details.html.pl#parent_connector','transparent volume remounting' ]}.</p>
-
-
- <a name="parent_connector"><h2><span class="constant">ParentConnector</span> volume remounter</h2></a>
-
- <p>The sandbox master component of this project has control of restarting
- its sandbox slaves containing the W32 filesystem. Target goal of
- <span class="constant">ParentConnector</span> component is to transparently
- provide persistent view of files and directories over the sandboxed slaves
- being restarted.</p>
-
- <p>In the case of read-only operations it would be simple as we could only
- save our state of currently opened filesystem objects with their read
- file/directory offset. Write operations can be handled as the read-only
- ones as long as all the operations are successful. In the case of W32
- filesystem crash we loose all the past write operations. If we would redo
- all the write operations we could very easily invoke the same crash.
- Therefore we write:</p>
-
- <blockquote class="command">
- <p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
- </blockquote>
-
- <p>message to syslog and refuse any further operations with this
- object.</p>
-
- @{[ doc_img 'dia/parent-connector','Parent Connector' ]}
-
- <p><span class="constant">HANDLE</span> represents W32 object open in
- existing W32 filesystem.<span class="constant">HANDLE</span> is created
- on-demand according to the saved state of the object (such as its
- pathname). Even the whole <span class="constant">VFS</span> sandbox slave
- is spawn on-demand if some object operation requests it.</p>
-
- <p>W32 filesystem crash can obviously occur at any moment - it generates
- @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
- @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
- <span class="constant">abort</span>. Successful filesystem unmount
- (even as the part of remount operation) must be first preceded by
- <span class="constant">detach</span> signal to close all existing
- W32 <span class="constant">HANDLE</span>s. After their close the filesystem
- gets the unmount requests. Only in the case all the close operations
- succeeded including the final filesystem unmount the signal
- <span class="constant">cease</span> can be activated to notify all the
- dirty (written) objects they are now clean. During this
- <span class="constant">cease</span> signal the project will also
- @{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
- underlying media.</p>
-
- <p>Objects never written remain in <span class="constant">clean</span>
- state and they can be transparently reopened even if W32 filesystem crash
- occurs.</p>
-
-
-HERE
-
-
-project::captive::doc::Macros->footer();
git://git.jankratochvil.net / www.jankratochvil.net.git / blobdiff