#! /usr/bin/perl
# 
# $Id$
# Captive project doc Details page Perl template.
# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; exactly version 2 of June 1991 is required
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA


package project::captive::doc::Details;
require 5.6.0;	# at least 'use warnings;' but we need some 5.6.0+ modules anyway
our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
our $CVS_ID=q$Id$;
use strict;
use warnings;

BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
use My::Web;
require CGI;
BEGIN { Wuse 'project::captive::doc::Macros'; }


My::Web->init(
		"__PACKAGE__"=>__PACKAGE__,
		"title"=>'Captive NTFS Developer Documentation: Implementation Details',
		"head_css"=>$doc_Macros_head_css,
		);
My::Web->heading();


print <<"HERE";


<h1>Implementation Details</h1>

	<a name="emulmeth"><h2>Choice of the Emulation Methods</h2></a>

		<p>The intent of the project was to get reliable read-write access to
		<span class="productname">NTFS</span> partition. There are several possible
		ways to achieve that:</p>

		<a name="emulmeth_vm"><h3>Virtualmachine Running the Original W32 Subsystem</h3></a>

			<p>Creating virtual-hardware PC and running the original W32 binaries
			including their boot-loader etc. Disk device access would be passed as
			virtual IDE disk (=hard disk drive). File access API would be implemented
			either by special escaping by some trapped instruction out of the
			virtualmachine while using W32 file access API or using the standard W32
			SMB (Server Message Block) network access through some virtual network
			card. The latter network access solution is almost the currently available
			possibility of running full-blown disk-sharing real
			<span class="productname">Microsoft Windows NT</span> inside virtual
			machine emulator such as <span class="productname">VMware</span>.</p>

			<p>pros: Full compatibility due to fully native codebase.</p>

			<p>cons: Hard to debug, missing documentation of NT booting internals,
			possible problems by different PC virtual-hardware than expected by NT,
			requirement of fully installed
			<span class="productname">Microsoft Windows NT</span> product.</p>

		<a name="method_ntoskrnl"><h3>&quot;ntoskrnl.exe&quot; Inside Virtual Address Space</h3></a>

			<p>This solution was chosen by the project. Binary filesystem driver and
			also <span class="fname">ntoskrnl.exe</span> binary file are required.
			Unfortunately <span class="fname">ntoskrnl.exe</span> expects a&nbsp;native
			PC virtual-hardware missing during regular UNIX user space process
			emulation, therefore such instructions must be trapped and emulated/ignored
			from case to case.</p>

			<p>Also the <a name="init_ntoskrnl">initialization code of <span
			class="fname">ntoskrnl.exe</span></a> is not executed by this project since
			it expects to get full PC hardware access privileges and thus some
			datastructures do not get initialized by it (need to be trapped later at
			runtime stage). Some of the missing initializations are solved by
			@{[ a_href 'APITypes.html.pl#functype_wrap','API functions wrapping' ]}.

			<p>pros: Lightweight, easier to debug.</p>

			<p>cons: Possible incompatible emulation of
			<span class="fname">ntoskrnl.exe</span> parts, missing documentation needed
			for the implementation.</p>

		<a name="emulmeth_fs"><h3>Filesystem Driver Inside Virtual Address Space</h3></a>

			<p>Unlike @{[ a_href 'Details.html.pl#method_ntoskrnl','previous method' ]} here we do not use
			even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
			W32 is <a name="native_ntoskrnl">emulated from the project source
			files</a>. <span class="fname">cdfs.sys</span> driver was successfuly ran
			in this manner in the former versions of this project but the possibility
			to run without <span class="fname">ntoskrnl.exe</span> was dropped since it
			had no licensing gains (you need the original
			<span class="productname">Microsoft Windows NT</span> files at least for
			the filesystem driver itself) and the emulation of undocumented parts
			reusable from <span class="fname">ntoskrnl.exe</span> binary was
			a&nbsp;pain.</p>

			<p>pros: Lightweight, easier to debug.</p>

			<p>cons: Possible incompatible emulation of the whole
			<span class="fname">ntoskrnl.exe</span>, its missing documentation.</p>


	<a name="apichoice"><h2>API Function Implementation Choices</h2></a>

		<p>During the initial point of the project development all the API
		functions were defined as unimplemented, of course. Any call of such
		unimplemented function is fatal and results in program termination. When we
		need to implement any required API function we have multiple choices to do
		so:
		@{[ a_href 'APITypes.html.pl#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
		@{[ a_href 'APITypes.html.pl#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
		@{[ a_href 'APITypes.html.pl#functype_native_reactos','Native implementation &ndash; $ReactOS' ]},
		@{[ a_href 'APITypes.html.pl#functype_native_wine','Native implementation &ndash; $Wine' ]}
		or
		@{[ a_href 'APITypes.html.pl#functype_native_libcaptive','Native implementation &ndash; project specific' ]}.
		<!-- a_href 'APITypes.html.pl#functype_undef','Undefined function' -->


	<a name="sandbox"><h2>Sandboxing of W32 Filesystem</h2></a>

		<p>The emulated W32 environment running the original W32 filesystem driver
		is separated from the rest of UNIX OS. It achieves the following goals:</p>

		<ul>
			<li><b>Restartable</b>: W32 driver can be restartde in clean state if it crashed</li>
			<li><b>Secure</b>: Malicious W32 code cannot affect the security of UNIX OS</li>
			<li><b>Stable</b>: Buggy W32 cannot crash any part of UNIX OS</li>
		</ul>

		<p>Sandboxing is provided with the following attributes:</p>
		
		<ul>
			<li>standalone UNIX process with separate memory space</li>
			<li>chroot(2) in empty directory to prevent any UNIX OS filesystem access</li>
			<li>setuid(2) to own user/group to prevent interaction with UNIX processes</li>
			<li>setrlimit(2) to limit system resources available for W32 environment</li>
			<li>the only connection with the UNIX OS by CORBA/ORBit RPC</li>
		</ul>

		<p>This security is almost the same as provided by
		emulated virtual machines such as
		@{[ a_href 'http://www.vmware.com/solutions/security.html','VMware' ]}.</p>

		@{[ doc_img 'dia/inheritance','Sandboxing Scheme' ]}

		<p>Project can be also used in non-sandboxed mode by
		<span class="command">--no-sandbox</span> option as it is easier to debug
		without CORBA/ORBit RPC. In this case the
		<span class="type">DirectorySlave</span>/<span class="type">FileSlave</span>
		options are used directly instead of their
		<span class="type">DirectoryParent</span>/<span class="type">FileParent</span>
		peers.</p>


	<a name="patched"><h2>&quot;patched&quot; vs. &quot;unpatched&quot; Libraries</h2></a>

		<p>Library is called <span class="constant">patched</span> if we require
		loading its original binary code file. Project needs to patch it to be able
		to trap all the function entry points. The only currently
		<span class="constant">patched</span> library of this project is
		<span class="fname">ntoskrnl.exe</span>.</p>

		<p>Library is called <span class="constant">unpatched</span> if no original
		binary code is needed since all of its functions are completely emulated by
		@{[ a_href 'APITypes.html.pl#functype_native','the native implementations' ]} of this project.
		The typical <span class="constant">unpatched</span> representative is
		<span class="fname">hal.dll</span> as it specializes on the hardware
		dependent code and therefore it must be completely replaced by this project
		running in the $gnulinux operating system environment. Early versions of
		this project had also full <span class="constant">unpatched</span>
		<a href="#native_ntoskrnl">native implementation of
		<span class="fname">ntoskrnl.exe</span></a> but it no longer applies.</p>

	<a name="mman"><h2>Memory Management</h2></a>

		<p>Original <span class="productname">Microsoft Windows NT</span>
		architecture uses two address space areas &ndash; user space and kernel space.
		User space is mapped in the range <span class="constant">0x00000000</span>
		to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
		range <span class="constant">0x80000000</span>
		(<span class="constant">KERNEL_BASE</span> in $ReactOS sources) to
		<span class="constant">0xFFFFFFFF</span>. All these virtual memory ranges
		represent addresses after their MMU (Memory Management Unit) mapping, of
		course. More discussion can be found in the
		<a href="http://www.microsoft.com/hwdev/platform/server/PAE/PAEmem.asp">description 
		by <span class="productname">Microsoft</span></a>.</p>

		<p>This project runs in the virtual address space used both for the UNIX
		user space process part and for the W32 kernel space. Therefore this
		project defines that W32 kernel runs in the whole range
		<span class="constant">0x00000000</span> to
		<span class="constant">0xFFFFFFFF</span> since there are no special mapping
		assumptions about the UNIX user space process mapping. No W32 user space
		exists in this project. Such approach also nullifies any special memory
		moving operations between W32 kernel space and W32 user space memory areas
		(such as <span class="function">MmSafeCopyToUser()</span>).</p>

	<a name="unicode"><h2>Unicode Strings and Characters</h2></a>

		<p>W32 platform uses 16-bit type <span class="type">wchar_t</span> while $gnulinux uses a
		32-bit one. This can be problem during GCC (GNU C&nbsp;Compiler)
		compilation of combination of native UNIX C&nbsp;sources (assuming 32-bit
		GCC with 32-bit <span class="type">wchar_t</span>) and
		$ReactOS C sources (assuming W32 compiler with 16-bit
		<span class="type">wchar_t</span>) for literal wide strings
		(C source file systax: <span class="command">L&quot;wstring&quot;</span>).
		Possibilities to solve this issue list:</p>

		<ul>
			<li>
				<p>Using <span class="constant">-fshort-wchar</span> GCC option and
				strictly differentiate between compilation of
				<span class="productname">ReactOS</span> code and UNIX code.</p>

				<p>pros: No source modifications needed, no runtime performance hit.</p>

				<p>cons: No type checking if some part of code has bad compilation
				flags, complicated way to completely split
				<span class="productname">ReactOS</span> and UNIX code.</p>
			</li>
			<li>
				<p>Wrap all <span class="productname">ReactOS</span> literal constants
				by some conversions function call (implemented as macro
				<span class="function">REACTOS_UCS2()</span> by this project).</p>

				<p>pros: Any forgotten/mistaken conversions are type-checked and warned
				during the compilation by GCC.</p>

				<p>cons: All compiled <span class="productname">ReactOS</span> sources
				files containing literal wide strings have to be wrapped/modified,
				performance hit by runtime string conversions.</p>

				<p>This solution was chosen to get the internal sanity checking
				benefit.</p>
			</li>
		</ul>

	<a name="binfmt"><h2>Supported Binary Formats</h2></a>

		<p>The native W32 binary format is identified as
		<span class="constant">PE-32</span> (Portable Executable 32-bit), such
		files have all the usual extensions such as
		<span class="fname">.sys</span>, <span class="fname">.exe</span>,
		<span class="fname">.dll</span> etc. <span class="constant">PE-32</span>
		loading support was already implemented by $ReactOS, its memory mapping
		specifics just had to be ported to $gnulinux environment by this project.
		This loading support does not (yet) cover importing of debug symbols from
		W32 <span class="fname">.PDB</span> (Program DataBase) files in $gnulinux
		ABI (Application Binary Interface) compatible way.</p>

		<p>This project also supports transparent loading of UNIX
		<span class="fname">.so</span> (Shared Object file) binary format. If you
		have W32 source files for some W32 library you can try to compile it by GCC
		to get the shared library with $gnulinux ABI compatible debug information
		(GCC option <span class="constant">-ggdb3</span> recommended). Beware of
		possible compilation problems as <span class="productname">Microsoft</span>
		C&nbsp;code expects <span class="constant">exception</span> handling to be
		supported by the compiler (definitely not the case of the plain C compiler
		of GCC) &mdash; all the exception catching code should be discarded as any
		@{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
		such driver is running in the scope of this project. You can use the
		following script of this project to compile W32 filesystem source files as
		UNIX <span class="fname">.so</span>:
		@{[ captive_srcfile 'src/w32-mod/ext2fsd.so-build.sh' ]}</p>
		
		<p>Be aware of some differences if you use
		<span class="constant">PE-32</span> binary format file vs.
		<span class="fname">.so</span> format file.
		<span class="constant">PE-32</span> use the appropriate W32 specific
		@{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
		<span class="fname">.so</span> must be completely compiled in the standard
		UNIX @{[ a_href 'CallType.html.pl#calltype_cdecl','cdecl call type semantics' ]}.
		@{[ a_href 'APITypes.html.pl#functype_native','Native function implementations' ]} do not need
		to be explicitely exported by <span class="fname">captivesym</span> as they
		are resolved automatically by the UNIX dynamic system linker. It may be
		surprising you will have to fix all such missing symbol exports if you
		advance during the development from the debugging
		<span class="fname">.so</span> file for the production version of the
		original <span class="constant">PE-32</span> binary file.</p>


	<a name="mounted_one"><h2>At Most One Mounted Filesystem</h2></a>

		<p>The project technically supports only one (exactly one...) mounted
		filesystem device and only one filesystem driver. There is nothing
		complicated to support multiple disks and multiple loaded filesystem
		modules but as they would share the address space it would only bring
		a&nbsp;possible complications during bug reports and the bug solving
		itself.  It was considered as a&nbsp;more sane way to support multiple W32
		mounted disks by completely separately running project instances in
		a&nbsp;different UNIX processes communicating from their sandboxes via
		@{[ a_href 'Details.html.pl#sandbox','CORBA sandbox interface' ]}. This sandboxing
		feature is not yet deployed although its code is already prepared.</p>

		<p>The project also does not support any state cleanup to be able to load
		filesystem&nbsp;<span class="constant">A</span>,
		cleanup&nbsp;<span class="constant">A</span> and load a different
		filesystem&nbsp;<span class="constant">B</span> in the same process address
		space. It complies with the preventions of the possible debugging
		complications as noted above. Despite this you still must call the function
		<span class="function">captive_shutdown()</span> to flush all the pending
		filesystem buffers to the disk. After calling
		<span class="function">captive_shutdown()</span> the process address space is
		no longer usable for any further project operations and the process is
		expected to be terminated in the manner compatible with its driving
		@{[ a_href 'Details.html.pl#sandbox','CORBA sandbox interface' ]} control master.</p>

		<p>Each sandbox executing the untrusted W32 binary filesystem driver code
		is connected through its
		@{[ a_href 'Details.html.pl#sandbox','CORBA sandbox interface' ]} at the point of upper
		layer <span class="constant">libcaptive</span>-specific filesystem API, at
		the point of the bottom layer of <span class="type">GIOChannel</span>
		device access and also for transfers of GLib logging
		messages/warnings/errors out of the sandbox to the user.</p>


	<a name="synchronous"><h2>Multithreading and Multiple Processors</h2></a>

		<p>W32 platform stands on its&nbsp;thorough architecture parallelism. It
		must lock all its objects to maintain coherence in presence of
		multithreading and multiple processors. Since the author of this project
		considers any parallel execution a serious obstacle for debugging the whole
		project architecture was designed to prevent any undeterministic behaviour.
		Therefore this projects always emulates uniprocessor
		<span class="productname">Microsoft Windows NT</span> kernel
		(<span class="constant">KeNumberProcessors</span> symbol is always 1),
		everything runs in the single initial thread/process and all the filesystem
		operations are performed as synchronous
			(&quot;synchronous&quot; by flags
			<span class="constant">FILE_SYNCHRONOUS_IO_ALERT</span>,
			<span class="constant">FO_SYNCHRONOUS_IO</span>,
			<span class="constant">IRP_SYNCHRONOUS_API</span>,
			<span class="constant">IRP_SYNCHRONOUS_PAGING_IO</span>,
			forced <span class="constant">TRUE</span> result of
			<span class="function">IoIsOperationSynchronous()</span>
			etc.).
		For several cases needed only by <span class="fname">ntfs.sys</span> there
		had to be supported asynchronous access
		(<span class="constant">STATUS_PENDING</span> return code) &ndash; parallel
		execution is emulated by GLib
		<span class="function">g_idle_add_full()</span> with
		<span class="function">g_main_context_iteration()</span> called during
		<span class="function">KeWaitForSingleObject()</span>.</p>
		Since there is a&nbsp;possibility a&nbsp;real W32 parallel threading would
		be yet needed in the future all the code that would be hit by W32
		multithreading capability is marked by
		<span class="constant">TODO:thread</span> comment.</p>

		<p>Multiple processors (SMP) support will never need to be implemented
		since uniprocessor W32 kernels apparently run the filesystem driver modules
		fine. As this project implements only the uniprocessor W32 kernel all the
		processor locking functions and structures such as
		<span class="constant">KSPIN_LOCK</span> etc. can be safely implemented as
		no-operations.</p>

		<p>Asynchronous callbacks registered for
		<span class="constant">IO_WORKITEM</span>s are passed as GLib idle
		functions by <span class="function">g_idle_add_full()</span>. Although they
		will probably never be executed during non-interactive project's batch
		executions it is the&nbsp;responsibility of W32 driver implementation to
		complete all the pending tasks before its W32 shutdown. Such W32 shutdown
		is done during cleanup of the project's&nbsp;execution by
		<span class="function">captive_shutdown()</span>.</p>

	<a name="paranoia"><h2>Paranoia Checks</h2></a>

		<p>A&nbsp;general approach of software projects development is to implement
		many internal sanity checks during the development stage but to produce the
		most optimized final release product without those debugging checks.</p>

		<p>Facilities for these practices can be seen in the standard
		C&nbsp;include files for example as function
		<span class="function">assert()</span> which gets disabled by the
		<span class="constant">NDEBUG</span> symbol used during the final optimized
		executable compilation. This project uses Gnome GLib messaging subsystem
		offering sanity checks discarded by symbols
		<span class="constant">G_DISABLE_ASSERT</span> and
		<span class="constant">G_DISABLE_CHECKS</span>.
		<span class="productname">Microsoft</span> also produces two versions of
		its products &ndash; regular customers use the &quot;free build&quot; (also
		called &quot;retail&quot;) while the programmers should develop their code
		on the &quot;checked build&quot; product releases.</p>

		<p>As this project will always run unknown binary code of proprietary W32
		filesystem drivers, the code can never be trusted. Such code even runs in
		the same unprotected address space as its controlling UNIX code. Since
		there is not enough documentation for the W32 components of the system and
		also such documentation is usually misleading it can never be considered as
		100% emulation. Even in the final releases all the sanity checks
		implemented in this project should remain active as all the project's code
		always interacts with unknown and untrusted W32 binaries.</p>

		<p><span class="productname">Microsoft Windows NT</span> code is written in
		a&nbsp;foolproof style as it accepts even invalid input values, and which
		it usually corrects. This makes long-term debugging a&nbsp;pain as it hides
		sources of problems. &quot;Checked build&quot; releases were probably
		designed to fix this flaw by strict consistency checks but it did not reach
		its goals as such checks are usually missing in the code.</p>

		<p>This project has strict consistency checks across all the code to make
		the debugging phase easy enough. Failed sanity check is not always
		a&nbsp;bug &ndash; sometimes it just means the real W32 binary code is more
		benevolent than it could be expected according to the documentation and
		such sanity check gets removed for the next version build. In other cases
		the failed sanity checks mean the execution path for some unexpected
		arguments combination was not yet implemented by this project. I may also
		mean a bug, of course...</p>

		<p>Last but not least &ndash; never miss a&nbsp;possible sanity check as its
		later removal is in an order of magnitude cheaper than an&nbsp;uncaught
		invalid assumption. Failed assertion is not always a&nbsp;bug although it
		has to be fixed, of course.</p>


	<a name="logfile"><h2>STATUS_LOG_FILE_FULL</h2></a>

		<p>After writing approx. 1MB of data on NTFS test partition NTFS driver
		returns for any further write requests
		<span class="constant">STATUS_LOG_FILE_FULL</status> error code.
		Apparently it is caused by the fact this project is
		@{[ a_href 'Details.html.pl#synchronous','single-threaded' ]} and it ignores the spawn
		of parallel journalling thread during <span class="fname">ntfs.sys</span>
		initialization.</p>

		<p>Fortunately <span class="fname">ntfs.sys</span> will clear its
		journalling log file during filesystem unmount. This project will therefore
		remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</status>
		is detected to workaround missing journalling thread.</p>

		<p>Similiar behaviour can be seen during write of compressed files &mdash;
		the file gets written uncompressed and its compression will proceed only
		during the final filesystem unmount.</p>

		<p>For these reasons it was mandatory to support
		@{[ a_href 'Details.html.pl#parent_connector','transparent volume remounting' ]}.</p>


	<a name="parent_connector"><h2><span class="constant">ParentConnector</span> volume remounter</h2></a>

		<p>The sandbox master component of this project has control of restarting
		its sandbox slaves containing the W32 filesystem. Target goal of
		<span class="constant">ParentConnector</span> component is to transparently
		provide persistent view of files and directories over the sandboxed slaves
		being restarted.</p>
		
		<p>In the case of read-only operations it would be simple as we could only
		save our state of currently opened filesystem objects with their read
		file/directory offset. Write operations can be handled as the read-only
		ones as long as all the operations are successful. In the case of W32
		filesystem crash we loose all the past write operations. If we would redo
		all the write operations we could very easily invoke the same crash.
		Therefore we write:</p>

			<blockquote class="command">
				<p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
			</blockquote>

		<p>message to syslog and refuse any further operations with this
		object.</p>

		@{[ doc_img 'dia/parent-connector','Parent Connector' ]}

		<p><span class="constant">HANDLE</span> represents W32 object open in
		existing W32 filesystem.<span class="constant">HANDLE</span> is created
		on-demand according to the saved state of the object (such as its
		pathname). Even the whole <span class="constant">VFS</span> sandbox slave
		is spawn on-demand if some object operation requests it.</p>

		<p>W32 filesystem crash can obviously occur at any moment - it generates
		@{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
		@{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
		<span class="constant">abort</span>. Successful filesystem unmount
		(even as the part of remount operation) must be first preceded by
		<span class="constant">detach</span> signal to close all existing
		W32 <span class="constant">HANDLE</span>s. After their close the filesystem
		gets the unmount requests. Only in the case all the close operations
		succeeded including the final filesystem unmount the signal
		<span class="constant">cease</span> can be activated to notify all the
		dirty (written) objects they are now clean. During this
		<span class="constant">cease</span> signal the project will also
		@{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
		underlying media.</p>

		<p>Objects never written remain in <span class="constant">clean</span>
		state and they can be transparently reopened even if W32 filesystem crash
		occurs.</p>


HERE


My::Web->footer();
