my $gnulinux='GNU/Linux';
-print vskip("10ex")."<h1 align=\"center\">!!! PRELIMINARY - TO BE UPDATED !!!</h1>\n".vskip("10ex");
+# FIXME:
+# Compatibility with NT4 etc. - just legal reasons.
+# FIXME:
+# name="cache_manager" Cache Manager
# FIXME:
-# Compatibility with NT4 etc. - just legal reasons.
+# name="sandbox"
print <<"HERE";
<p>Currently there is no possibility to any of the available $freespeech
($freespeech used in the following text in the meaning of
- "<a href="http://www.gnu.org/philosophy/free-sw.html">free as in speech</a>")
+ "@{[ a_href 'http://www.gnu.org/philosophy/free-sw.html','free as in speech' ]}")
operating systems to reliably write to the most common disk partition
- filesystem type - <span class="productname">Microsoft NTFS</span>. It would
+ filesystem type – <span class="productname">Microsoft NTFS</span>. It would
be already supported a long time ago but there is no proper documentation of
<span class="productname">NTFS</span> filesystem data structures available.
Since <span class="productname">Microsoft</span> corporation continues in its
read-write part of the access would require much better
knowledge of the <span class="productname">NTFS</span> data structures.
Currently only rewriting of already existing file data blocks is supported
- by $LinuxNTFS - no file creation, no file deletion, no directory operations etc.
+ by $LinuxNTFS — no file creation, no file deletion, no directory operations etc.
Also any future versions of <span class="productname">NTFS</span> filesystem
would require another major reverse engineering effort.</p>
<h1>Challenges of the Project</h1>
<p>The <a name="NTFSgoal">ultimate goal</a> of this project is definitely the
- free implementation of <a href="#reliability">reliable</a> read-write <span
+ free implementation of @{[ a_href '#reliability','reliable' ]} read-write <span
class="productname">NTFS</span> filesystem driver. This project chose to
solve this problem in the style of $Wine project by using the original binary
<span class="fname">ntfs.sys</span> and emulating all the required layers of
insufficient documentation of API between filesystem driver
(<span class="fname">ntfs.sys</span>) and the
<span class="productname">Microsoft Windows NT</span>
- ("<a href="http://mail.gnu.org/archive/html/libtool/2000-09/msg00000.html">W32</a>"
+ ("@{[ a_href 'http://mail.gnu.org/archive/html/libtool/2000-09/msg00000.html','W32' ]}"
in the following text) kernel <span class="fname">ntoskrnl.exe</span>. Note
that this API is a different than the one being used in the $Wine project
since <span class="productname">Wine</span> implements only the user space
<table align="center" border="1">
<tr>
- <th><a href="#guestosnote">Guest-OS</a></th>
- <th><a href="#hostosnote" >Host-OS</a ></th>
+ <th>@{[ a_href '#guestosnote','Guest-OS' ]}</th>
+ <th>@{[ a_href '#hostosnote' ,'Host-OS' ]}</th>
<th>Implements</th>
<th>W32 kernel library</th>
</tr>
<dl>
<a name="guestosnote"><dt>Guest-OS</dt></a>
- <dd><a href="http://www.vmware.com/support/reference/common/glossary/#guestos">Guest OS</a>:
+ <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#guestos','Guest OS' ]}:
An operating system that runs inside a virtual machine.</dd>
<a name="hostosnote" ><dt>Host OS</dt></a>
- <dd><a href="http://www.vmware.com/support/reference/common/glossary/#hostos" >Host OS</a>:
+ <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#hostos' ,'Host OS' ]}:
An operating system that runs on the host machine.</dd>
</dl>
<p>While $ReactOS provides the necessary W32 kernel subsystem emulation
- code we also need to run such <a href="#guestosnote">Guest-OS</a> in the <a
- href="#hostosnote">Host-OS</a> $gnulinux. Initially it was planned to
+ code we also need to run such @{[ a_href '#guestosnote','Guest-OS' ]} in the
+ @{[ a_href '#hostosnote','Host-OS' ]} $gnulinux. Initially it was planned to
extend $Wine with the W32 kernel space emulation functionality but
fortunately <span class="author">Steven Edwards</span> pointed to the $ReactOS
which better suits the needs of this project by its already implemented W32
<p>The most serious problem of $ReactOS is its dependence on the direct
<span class="constant">i386</span> hardware instead of some
- <a href="#hostosnote">Host-OS</a> as required by the goals of this project.
+ @{[ a_href '#hostosnote','Host-OS' ]} as required by the goals of this project.
W32 is designed to be hardware-independent using its
<span class="fname">hal.dll</span>. Unfortunately $ReactOS does not follow
this design and thus there are needed various patches and replaces of its
various parts and its hardware-dependent code. Despite it $ReactOS code
base still made a big asset for this project.</p>
-
+ <p class="stuff">... and @{[ a_href 'http://www.reactos.com/','ReactOS' ]} cannot run on Linux!<br />
<span class="author">Casper Hornstrup</span> enlightened such functions
calling conventions have to be differentiated as
<span class="fname">ntdll.dll</span> lives in the user space (low address
- space -- below <span class="constant">0x80000000</span>) and
+ space – below <span class="constant">0x80000000</span>) and
<span class="fname">ntoskrnl.exe</span> in the kernel space (high address
- space -- above <span class="constant">0x80000000</span>). Although they
+ space – above <span class="constant">0x80000000</span>). Although they
contain slightly different set of symbols (functions)
<span class="fname">ntdll.dll</span> still can be considered as a user
space interface to the kernel space implementation by
no plans to ever extend the project's crossplatformity beyond the
<span class="constant">i386</span> processor
(<span class="constant">i386</span> used here as
- <a href="http://www.intel.com/">Intel</a> architecture covering 32-bit
+ @{[ a_href 'http://www.intel.com/','Intel' ]} architecture covering 32-bit
processors compatible with <span class="constant">i386</span>,
<span class="constant">i486</span>, ...).</p>
unimplemented function is fatal and results in program termination. When we
need to implement any required API function we have multiple choices to do
so:
- <a href="#functype_pass">Direct pass to original
- <span class="fname">ntoskrnl.exe</span></a>,
- <a href="#functype_wrap">Wrap of the original
- <span class="fname">ntoskrnl.exe</span> function</a>,
- <a href="#functype_native_reactos">Native implementation -- $ReactOS,
- <a href="#functype_native_wine">Native implementation -- $Wine
+ @{[ a_href '#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
+ @{[ a_href '#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
+ @{[ a_href '#functype_native_reactos','Native implementation – $ReactOS' ]},
+ @{[ a_href '#functype_native_wine','Native implementation – $Wine' ]}
or
- <a href="#functype_native_libcaptive">Native implementation
- -- project specific</a>.
- <!-- a href="#functype_undef" Undefined function /a -->
+ @{[ a_href '#functype_native_libcaptive','Native implementation – project specific' ]}.
+ <!-- a_href '#functype_undef','Undefined function' -->
<h2>"patched" vs. "unpatched" Libraries</h2>
<p>Library is called <span class="constant">unpatched</span> if no original
binary code is needed since all of its functions are completely emulated by
- <a href="#functype_native">the native implementations</a> of this project.
+ @{[ a_href '#functype_native','the native implementations' ]} of this project.
The typical <span class="constant">unpatched</span> representative is
<span class="fname">hal.dll</span> as it specializes on the hardware
dependent code and therefore it must be completely replaced by this project
<h2>Memory Management</h2>
<p>Original <span class="productname">Microsoft Windows NT</span>
- architecture uses two address space areas - user space and kernel space.
+ architecture uses two address space areas – user space and kernel space.
User space is mapped in the range <span class="constant">0x00000000</span>
to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
range <span class="constant">0x80000000</span>
possible compilation problems as <span class="productname">Microsoft</span>
C code expects <span class="constant">exception</span> handling to be
supported by the compiler (definitely not the case of the plain C compiler
- of GCC) --- all the exception catching code should be discarded as any
- <a href="#exception_fatal">generated exceptions are always fatal</a> when
+ of GCC) — all the exception catching code should be discarded as any
+ @{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
such driver is running in the scope of this project. You can use the
following script of this project to compile W32 filesystem source files as
UNIX <span class="fname">.so</span>:
<span class="constant">PE-32</span> binary format file vs.
<span class="fname">.so</span> format file.
<span class="constant">PE-32</span> use the appropriate W32 specific
- <a href="#calltype">cdecl/stdcall/fastcall call types</a>,
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
<span class="fname">.so</span> must be completely compiled in the standard
- UNIX <a href="#calltype_cdecl">cdecl call type semantics</a>.
- <a href="#functype_native">Native function implementations</a> do not need
+ UNIX @{[ a_href '#calltype_cdecl','cdecl call type semantics' ]}.
+ @{[ a_href '#functype_native','Native function implementations' ]} do not need
to be explicitely exported by <span class="fname">captivesym</span> as they
are resolved automatically by the UNIX dynamic system linker. It may be
surprising you will have to fix all such missing symbol exports if you
documented at all and the others are documented insufficiently for a their
possibly needed reimplementation from scratch. Documentation being
consulted primarily consists of
- <span class="productname"><a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/kmarch/hh/kmarch/kmhdr_6enb.asp">MSDN (Microsoft Developer Network) Kernel-Mode Driver Architecture: Windows DDK</a></span>
+ <span class="productname">@{[ a_href 'http://msdn.microsoft.com/library/default.asp?url=/library/en-us/kmarch/hh/kmarch/kmhdr_6enb.asp','MSDN (Microsoft Developer Network) Kernel-Mode Driver Architecture: Windows DDK' ]}</span>
documentation and also various other 3rd party documentation resources such as
- <span class="productname"><a href="http://www.osr.com/ntinsider/1996/cacheman.htm">The NT Cache Manager Description</a></span>,
- <span class="productname"><a href="http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864">Learn About NT's File-system Cache</a></span>,
- <span class="productname"><a href="http://www.ntfsd.org/archive/">NT File System Developers mailing list archives</a></span>
+ <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
+ 'The NT Cache Manager Description' ]}</span>,
+ <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
+ 'Learn About NT'."'".'s File-system Cache' ]}</span>,
+ <span class="productname">@{[ a_href 'http://www.ntfsd.org/archive/',
+ 'NT File System Developers mailing list archives' ]}</span>
including various
- <a href="http://www.google.com/search?q=site%3Amicrosoft.com">fulltext searches</a>
+ @{[ a_href 'http://www.google.com/search?q=site%3Amicrosoft.com','fulltext searches' ]}
through Internet from case to case.</p>
<p>Sometimes no sufficient documentation was found and some code behaviour
and primarily
<span class="fname">ntfs.sys</span>.
Up to now the code was disassembled by
- <span class="productname"><a href="http://www.simtel.net/pub/pd/29498.html">IDA Freeware</a></span>
+ <span class="productname">@{[ a_href 'http://www.simtel.net/pub/pd/29498.html','IDA Freeware' ]}</span>
and by
<span class="productname">dumpbin.exe</span> of
<span class="productname">Microsoft Visual Studio</span>.
<p>This is (the only?) tool able to debug filesystem drivers incl.
<span class="fname">ntfs.sys</span>. You will need two computers running
- <span class="productname">Microsoft Windows</span> - one computer will run
+ <span class="productname">Microsoft Windows</span> — one computer will run
<span class="productname">WinDbg</span> while the other one will be
frozen in remote Windows NT kernel debug mode. It does not matter which
<span class="productname">Microsoft Windows</span> version will be run
on the <span class="productname">WinDbg</span> side.</p>
<p>The most easy way to setup two computers is to use commercial
- <span class="productname"><a href="http://www.vmware.com/download/workstation.html">VMware Workstation</a></span>
+ <span class="productname">@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</span>
where you can run two virtual machines simultaneously on single PC
hardware and you can connect them by a virtual serial port provided by
<span class="productname">VMware</span>.</p>
$freebeer update packages for its
<span class="productname">Microsoft Windows</span> products called
<span class="productname">Service Packs</span>; the latest one is
- <span class="productname"><a href="http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp">Microsoft Windows XP Service Pack 1a</a></span>.</p>
+ <span class="productname">@{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp','Microsoft Windows XP Service Pack 1a' ]}</span>.</p>
<p>This downloadable file contains the full versions of the essential
files needed for the current stage of this product:
<p>Most of the work of this project is located in the single box called
"<span class="constant">libcaptive</span>" located in the center
of the scheme. This component implements the core W32 kernel API by
- <a href="#functype">various methods described in this document</a>.
+ various methods described in this document.
The "<span class="constant">libcaptive</span>" box cannot be
- further dissected as it is just an implementation of a set of API
- functions. It could be separated to several subsystems such as the Cache
- Manager, Memory Manager, Object Manager, Runtime Library, I/O Manager
+ further dissected as it is just an implementation of a set of
+ @{[ a_href 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/src/libcaptive/ke/exports.captivesym?rev=HEAD',
+ 'API functions' ]}.
+ It could be separated to several subsystems such as the
+ @{[ a_href '#cache_manager','Cache Manager' ]},
+ Memory Manager, Object Manager, Runtime Library, I/O Manager
etc. but they have no interesting referencing structure.</p>
<p>As this project is in fact just a filesystem implementation every
story must begin at the device file and end at the filesystem operations
interface. The unified suppported interfaces are
- <span class="productname"><a href="http://developer.gnome.org/doc/API/2.0/glib/">GLib</a></span>
+ <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
(the most low level portability, data-types and utility library for Gnome)
<span class="type">GIOChannel</span> (for the device access) and the custom
- <span class="constant">libcaptive</span> filesystme API. Each of these ends
+ <span class="constant">libcaptive</span> filesystem API. Each of these ends
can be connected either to some direct interface (such as the
- <span class="constant">captive-cmdline</span> client) or it can connected
- as a general $GnomeVFS filter. $GnomeVFS offers nice filter interface on
+ <span class="constant">captive-cmdline</span> client),
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
+ or as a general $GnomeVFS filter.
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
+ most cases as it offers standard filesystem interface by Linux kernel.
+
+ You can also use $GnomeVFS as it offers nice filter interface on
the UNIX user-privileges level for transparent operation with archives and
network protocols. This filter interface was used by this project to turn
- the device reference such as <span class="fname">/dev/hda3</span> or
- <span class="fname">/dev/discs/disc0/part3</span> to the fully accessible
+ the device reference such as <span class="fname">/dev/hda3</span> or <span
+ class="fname">/dev/discs/disc0/part3</span> to the fully accessible
filesystem (pretending being an "archive" in the device
reference). This device access can be specified by $GnomeVFS URLs such as:
- <span class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
-
+ <span
+ class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
+
+ <span class="constant">captive-bug-replay</span> serves just for debugging
+ purposes — you can 'replay' existing
+ <span class="fname">file.captivebug.xml.gz</span> automatically being
+ generated during W32 filesystem failure. This bugreport file will contain
+ all the touched data blocks of the device used in the moment of the
+ failure. <span class="constant">captive-bug-replay</span> will therefore
+ emulate internal virtual writable device out of these bugreported data.
+
<p>If the passed device reference is requested by the user to be accessed
either in <span class="dashdash">--ro</span> (read-only) mode or in the
<span class="dashdash">--rw</span> (full read-write) mode there are no
- further device layers needed. Just in the case of
- <span class="dashdash">--blind</span> mode another layer is involved to
- emulate read-write device on top of the real read-only device by the method
- of non-persistent memory buffering of all the possible write requests.</p>
+ further device layers needed. Just in the case of <span
+ class="dashdash">--blind</span> mode another layer is involved to emulate
+ read-write device on top of the real read-only device by the method of
+ non-persistent memory buffering of all the possible write requests.</p>
+
+ <span class="constant">sandbox commit buffer</span> is involved only in the
+ case @{[ a_href '#sandbox','sandboxing feature' ]} is active. It will
+ buffer any writes to the device during the sandbox run to prevent
+ filesystem damage if the driver would fail in the meantime. If the
+ filesystem gets finally successfully unmounted this sandbox buffer can be
+ <a name="safe_flush">safely flushed</a>
+ to its underlying physical media. The buffer will be dropped
+ in the case of filesystem failure, of course. The filesystem should be
+ unmounted from time to time — it can be transparently unmounted and mounted
+ by <span class="command">commit</span> of
+ <span class="constant">captive-cmdline</span> custom client. Currently you
+ cannot force remounting when using
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
+ but it will be remounted after approx each 1MB data written automatically
+ due to @{[ a_href '#log_file_full','NTFS log file full' ]}.
+
+ Now we need to transparently
+ @{[ a_href 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/src/libcaptive/sandbox/sandbox.idl?rev=HEAD',
+ 'connect' ]}
+ the device interface of <span class="type">GIOChannel</span> type through
+ @{[ a_href '#sandbox','CORBA/ORBit' ]} to the sandboxed slave.
<p>Such device is still only a UNIX style GLib <span
- class="type">GIOChannel</span> type at this point. As we need to supply it
+ class="type">GIOChannel</span> type at this point. As we need to supply it
to the W32 filesystem driver we must convert it to the W32 I/O Device
with its capability of handling <span class="type">IRP</span>
(<span class="constant">I/O Request Packet</span>; structure holding the
requests from its upper W32 filesystem driver. Such W32 I/O Device can
represent either <span class="type">CD-ROM</span> or
<span class="type">disk</span> device type as different W32 filesystem
- drivers require different media types:</p>
-
- <h3>cdfs.sys</h3>
-
- <p><span class="type">CD-ROM</span> filesystem runs just on the
- <span class="constant">FILE_DEVICE_CD_ROM_FILE_SYSTEM</span> device type.
- Use <span class="dashdash">--cdrom</span> option of this project for
- <span class="fname">cdfs.sys</span>.</p>
-
- <h3>fastfat.sys</h3>
-
- <p><span class="type">FAT</span> filesystem supports both the (expected)
- <span class="constant">FILE_DEVICE_DISK_FILE_SYSTEM</span> device type
- but it also supports the reading of
- <span class="constant">FILE_DEVICE_CD_ROM_FILE_SYSTEM</span> devices as
- you can use <span class="type">FAT</span> filesystem on <span
- class="type">CD-ROM</span> media in W32 environment. It is recommended to
- use <span class="dashdash">--disk</span> option of this project for
- <span class="fname">fastfat.sys</span>.</p>
-
- <h3>ext2fsd.sys</h3>
-
- <p><span class="type">ext2</span> filesystem supports just the
- <span class="constant">FILE_DEVICE_DISK_FILE_SYSTEM</span> device type.
- Use <span class="dashdash">--disk</span> option of this project for
- <span class="fname">ext2fsd.sys</span>.</p>
-
- @{[ vskip("3ex") ]}
+ drivers require different media types — currently only
+ <span class="fname">cdfs.sys</span> requires
+ <span class="type">CD-ROM</span> type.</p>
<p>W32 media I/O Device is accessed from the W32 filesystem driver.
The filesystem driver itself always creates volume object by
<p>The filesystem driver is called by the core W32 kernel implementation of
<span class="constant">libcaptive</span> in
- <a href="#synchronous">synchronous way</a> in single-shot manner instead of
+ @{[ a_href '#synchronous','synchronous way' ]} in single-shot manner instead of
the several reentrancies while waiting for the disk I/O completions as can
be seen in the original
<span class="productname">Microsoft Windows NT</span>.
This single-shot synchronous behaviour is possible since all the needed
resources (disk blocks etc.) can be always presented as instantly ready as
- their acquirement is solved by <a href="hostosnote">Host-OS</a> outside of
- the W32 emulated <a href="guestosnote">Guest-OS</a> environment.</p>
+ their acquirement is solved by @{[ a_href 'hostosnote','Host-OS' ]} outside of
+ the W32 emulated @{[ a_href 'guestosnote','Guest-OS' ]} environment.
+ For several cases needed only by <span class="fname">ntfs.sys</span>
+ there had to be supported asynchronous access — parallel execution
+ is emulated by GLib <span class="function">g_idle_add_full()</span>
+ with <span class="function">g_main_context_iteration()</span> called during
+ <span class="function">KeWaitForSingleObject()</span>.</p>
<p><span class="constant">libcaptive</span> offers the W32 kernel
filesystem API to the upper layers. This is still not the API the common
<p>As it would be very inconvenient to use the legacy, bloated and UNIX
style unfriendly W32 kernel filesystem API this project offers its own
- <a href="#client_interface">custom filesystem API interface</a> inspired by
+ @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
This interface is supposed to be easily utilized by
<a href="#client_interface_customapp">a custom application accessing
the W32 filesystem driver</a>.</p>
+ <p>@{[ a_href '#sandbox','CORBA/ORBit' ]} hits us again – we need to
+ @{[ a_href 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/src/libcaptive/sandbox/sandbox.idl?rev=HEAD',
+ 'translate' ]}
+ the @{[ a_href '#client_interface','custom filesystem API interface' ]}
+ out of the sandboxed slave to the UNIX space.</p>
+
+ <p><span class="constant">captive sandbox master</span> provides the
+ functionality of covering any possible sandboxed slave restarts and its
+ communication. It is also capable of
+ <a name="demultiplexing_master">demultiplexing single API operations</a>
+ to multiple its connected sandbox slaves in transparent way
+ as each of them handles
+ @{[ a_href '#mounted_one','just one filesystem device' ]}.</p>
+
<p>The rest of the story is not much special for this project since this is
a common UNIX problem how to offer user space implemented UNIX filesystem
as a generic system filesystem (as those are usually implemented only as
- the components od UNIX kernel). The most thin implementation would be to
- implement <FIXME:LUFS><a href="#fuse_interface">FUSE \bookcitation{FUSE}
- (Filesystem in Userspace project for $gnulinux implemented by its own
- filesystem code for Linux kernel)
- interface</a> for the purpose but such feature is not yet implemented.
- Currently this project implements
- <a href="#offered_gnomevfs">Gnome-VFS interface</a> allowing its filesystem
- access even without any involvement of UNIX kernel from any
- $GnomeVFS aware client application (such as
- <span class="fname">gnome-vfs/tests/test-shell</span>).
- This <a href="#offered_gnomevfs">Gnome-VFS interface</a> connects the data
- flow of this project in two points - both as the lowest layer device image
- source and also as the upper layer for the filesystem operation
- requests.</p>
-
- <p>That's all folks!</p>
+ the components od UNIX kernel).</p>
+
+ <p>The filesystem service can be offered in several ways:</p>
+
+ <dl>
+ <dt>Custom client</dt>
+ <dd>
+ <p>One possibility would be to write
+ <a name="client_interface_customapp">a custom client application</a>
+ for this project such as file manager or a shell. Although it
+ would implement the most appropriate user interface to the set of
+ functions offered by this project (and W32 filesystem API) it has the
+ disadvantage of special client software. Appropriate client is provided
+ by this project as:
+ <span class="fname">src/client/cmdline/cmdline-captive</span></p>
+ </dd>
+
+ <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
+ <dd>
+ <p>The most usable interface is the
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
+ by <span class="constant">liblufs-captivefs</span>.
+ As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
+ already assigns separate process for each filesystem mount the
+ @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
+ is not utilized in this case.</p>
+
+ <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
+ needs multiple operating threads (each UNIX kernel operation needs
+ one free lufsd slot/thread to not to fail immediately).
+ As <span class="constant">libcaptive</span> is
+ @{[ a_href '#synchronous','single-threaded' ]} all the operations
+ get always synchronized by
+ <span class="constant">liblufs-captivefs</span>
+ before their pass over to <span class="constant">libcaptive</span>.</p>
+ </dd>
+
+ <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
+ <dd>
+ <p>This client allowing its filesystem access even without any
+ involvement of UNIX kernel from any $GnomeVFS aware client application
+ (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
+ This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
+ data flow of this project in two points — both as the lowest layer
+ device image source and also as the upper layer for the filesystem
+ operation requests.</p>
+ </dd>
+ </dl>
+
+ <p>Unimplemented and deprecated methods for providing filesystem
+ service:</p>
+
+ <dl>
+ <dt>W32 filesystem in UNIX OS kernel</dt>
+ <dd>
+ <p>The real UNIX OS filesystem implementation must be completely
+ implemented inside the hosting OS kernel. This requires special coding
+ methods with limited availability of coding features and libraries.
+ Also it would give the full system control to the untrusted W32
+ filesystem driver code with possibly fatal consequences of yet
+ unhandled W32 emulation code paths. It would benefit from the best
+ execution performance but this solution was never considered a real
+ possibility.</p>
+ </dd>
+
+ <dt>Custom NFS server</dt>
+ <dd>
+ <p>The common approach
+ <a name="offered_NFS">of filesystem implementations</a>
+ outside UNIX OS kernel were custom NFS servers usually running on the
+ same machine as the NFS-connected client as such NFS server is usually
+ an ordinary UNIX user space process. It would be possible to implement
+ this project as a custom NFS server but the NFS protocol itself
+ has a lot of fundamental flaws and complicated code for backward
+ compatibility.</p>
+ </dd>
+ </dl>
+
<a name="mounted_one"><h2>At Most One Mounted Filesystem</h2></a>
itself. It was considered as a more sane way to support multiple W32
mounted disks by completely separately running project instances in
a different UNIX processes communicating from their sandboxes via
- <a href="#todo_sandbox">CORBA sandbox interface</a>. This sandboxing
+ @{[ a_href '#sandbox','CORBA sandbox interface' ]}. This sandboxing
feature is not yet deployed although its code is already prepared.</p>
<p>The project also does not support any state cleanup to be able to load
<span class="function">captive_shutdown()</span> the process address space is
no longer usable for any further project operations and the process is
expected to be terminated in the manner compatible with its driving
- <a href="#todo_sandbox">CORBA sandbox interface</a> control master.</p>
+ @{[ a_href '#sandbox','CORBA sandbox interface' ]} control master.</p>
<p>Each sandbox executing the untrusted W32 binary filesystem driver code
is connected through its
- <a href="#todo_sandbox">CORBA sandbox interface</a> at the point of upper
+ @{[ a_href '#sandbox','CORBA sandbox interface' ]} at the point of upper
layer <span class="constant">libcaptive</span>-specific filesystem API, at
the point of the bottom layer of <span class="type">GIOChannel</span>
device access and also for transfers of GLib logging
it expects to get full PC hardware access privileges and thus some
datastructures do not get initialized by it (need to be trapped later at
runtime stage). Some of the missing initializations are solved by
- <a href="#functype_wrap">API functions wrapping</a>.
+ @{[ a_href '#functype_wrap','API functions wrapping' ]}.
<p>pros: Lightweight, easier to debug.</p>
<h2>Filesystem Driver Inside Virtual Address Space</h2>
- <p>Unlike <a href="#method_ntoskrnl">previous method</a> here we do not use
+ <p>Unlike @{[ a_href '#method_ntoskrnl','previous method' ]} here we do not use
even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
W32 is <a name="native_ntoskrnl">emulated from the project source
files</a>. <span class="fname">cdfs.sys</span> driver was successfuly ran
functionality. Currently implemented functionality statistics are provided
below:</p>
- <FIXME:numbers>
<table border="1" align="center">
<tr><th>Function type </th><th>Items</th><th>Portion</th></tr>
- <tr><td><a href="#functype_pass">pass</a> </td><td> 46</td><td> 21%</td></tr>
- <tr><td><a href="#functype_wrap">wrap</a> </td><td> 1</td><td> 0%</td></tr>
- <tr><td><a href="#functype_native_reactos">native-ReactOS</a></td><td> 94</td><td> 43%</td></tr>
- <tr><td><a href="#functype_native_libcaptive">native-own</a> </td><td> 79</td><td> 36%</td></tr>
+ <tr><td>@{[ a_href '#functype_pass','pass' ]} </td><td> 81</td><td> 26%</td></tr>
+ <tr><td>@{[ a_href '#functype_wrap','wrap' ]} </td><td> 2</td><td> 0%</td></tr>
+ <tr><td>@{[ a_href '#functype_native_reactos','native-ReactOS' ]}</td><td> 113</td><td> 36%</td></tr>
+ <tr><td>@{[ a_href '#functype_native_libcaptive','native-own' ]} </td><td> 116</td><td> 38%</td></tr>
<caption>Function Implementation Types Statistics</caption>
</table>
contain already prepared content at the runtime. There is a problem
with <span class="constant">patched</span> libraries where it is necessary
to also fully implement the data symbol as
- <a href="#functype_native">native implementation</a> since there is no
- possibility to <a href="#functype_pass">pass</a> the data symbol instead of
+ @{[ a_href '#functype_native','native implementation' ]} since there is no
+ possibility to @{[ a_href '#functype_pass','pass' ]} the data symbol instead of
the original W32 data location and therefore there will be two instances of
such data variable place. As there will be also the uncaught references for
such W32 data location from the <span class="constant">patched</span>
<p>All the exporting magic is handled by custom script
<span class="fname">captivesym</span> processing the definition file
- <FIXME:span class="fname">src/libcaptive/ke/exports.captivesym</span> to produce
- the intermediate relaying code
- <FIXME:span class="fname">src/libcaptive/ke/exports.c</span>. For details of the
+ <span class="fname">@{[ a_href
+ 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/src/libcaptive/ke/exports.captivesym?rev=HEAD',
+ 'src/libcaptive/ke/exports.captivesym' ]}</span>
+ to produce the intermediate relaying code
+ <span class="fname">src/libcaptive/ke/exports.c</span>. For details of the
<span class="fname">captivesym</span>-specific source file syntax please
- see its documentation: <FIXME:span class="fname">doc/captivesym-pod.html</span>
+ see its documentation:
+ <span class="fname">@{[ a_href
+ $W->{"top_dir"}.'/project/Pod2Html.html.pl?cvs=priv/captive/src/libcaptive/ke/captivesym.pl',
+ 'src/libcaptive/ke/captivesym.pl' ]}</span>
<a name="functype_pass"><h3>Direct Pass to Original "ntoskrnl.exe"</h3></a>
<a name="functype_pass_fromunix"><h4>Pass from UNIX Code</h4></a>
<p>Control flow begins in some standard UNIX code. Such code is always
- using <a href="#calltype_cdecl">cdecl call type</a> for all its
+ using @{[ a_href '#calltype_cdecl','cdecl call type' ]} for all its
intracalls. <a href="#functype_native_reactos">Native functions
compiled from <span class="productname">ReactOS</span> sources</a> use
- their own <a href="#calltype">cdecl/stdcall/fastcall</a> declarations
+ their own @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} declarations
but these call type modifications are discarded during compilation for
this project by the <span class="constant">LIBCAPTIVE</span>
symbol.</p>
from the generated UNIX jump table. Such relay will debug dump the
passed arguments and finally pass the control to the original W32
function code in the proper call type
- <a href="#calltype">cdecl/stdcall/fastcall</a> for a given
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} for a given
function.</p>
<p>Original W32 code entry point is always trapped by a breakpoint
patch another breakpoint in place of
<span class="productname">slot #2</span>.
During the <span class="productname">slot #2</span> breakpoint
- invocation the operation will be reverted - the breakpoint will be put
+ invocation the operation will be reverted — the breakpoint will be put
to <span class="productname">slot #1</span> again and the instruction
of <span class="productname">slot #2</span> will be restored to be able
to continue the execution of the function.</p>
<p>W32 function will finish in its specific
- <a href="#calltype">cdecl/stdcall/fastcall call type</a>, the control
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}, the control
will return to the UNIX jump table relay which will debug dump the
return value and it will finally pass the control back to the UNIX
caller in the standard UNIX
- <a href="#calltype_cdecl">cdecl call type</a>.</p>
+ @{[ a_href '#calltype_cdecl','cdecl call type' ]}.</p>
@{[ doc_img 'fig/functype_patched_pass_fromunix',
'Function Type: <span class="constant">pass</span> from UNIX Code' ]}
<a name="functype_pass_fromw32"><h4>Pass from W32 Code</h4></a>
<p>This function type is similiar to the
- <a href="#functype_pass_fromunix">previous one</a> with the exception
+ @{[ a_href '#functype_pass_fromunix','previous one' ]} with the exception
of more complicated entry point. Unfortunately W32 libraries call their
own functions directly, using the <span class="instruction">call</span>
instructions without any patchable jump table. Even the
according to the relocation table record as such library intra-call
instruction has no relocation due to its relative argument offset on
<span class="constant">i386</span>. This time the double-breakpoint
- mechanism <a href="#functype_pass_fromunix">described above</a> gets
+ mechanism @{[ a_href '#functype_pass_fromunix','described above' ]} gets
handy since it will catch the entry point when the function gets
called. <span class="constant">SIGSEGV</span> handler gets invoked by
the <span class="instruction">hlt</span> instruction and it will
<p>The jump table relay used for the callers from W32 code is
a different one than the relay being used for the callers
- <a href="#functype_pass_fromunix">from UNIX code</a>. UNIX code always
- uses relay with external <a href="#calltype_cdecl">cdecl call type</a>
+ @{[ a_href '#functype_pass_fromunix','from UNIX code' ]}. UNIX code always
+ uses relay with external @{[ a_href '#calltype_cdecl','cdecl call type' ]}
but in this case a relay with the appropriate
- <a href="#calltype">cdecl/stdcall/fastcall call type</a> is used.</p>
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} is used.</p>
@{[ doc_img 'fig/functype_patched_pass_fromw32',
'Function Type: <span class="constant">pass</span> from W32 Code' ]}
<p>The code control flow has no special hardcore features since it is
very similiar to <a href="#functype_pass_fromunix">the direct pass to
W32 function from UNIX code</a>. All the wrapping is done in the
- standard UNIX <a href="#calltype_cdecl">cdecl call type</a> manner.
- Jump table debug dumping relays are provided twice - the
+ standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.
+ Jump table debug dumping relays are provided twice — the
"outer" one to trace the parameters from the function caller
and the "inner" one to trace the call from the wrapper to the
original W32 code. The "inner" relay also calls the W32 code
<span class="constant">through_w32_func</span> field of this function
description already set is done from the "inner" jump table
relay with the appropriate
- <a href="#calltype">cdecl/stdcall/fastcall call type</a>.</p>
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}.</p>
@{[ doc_img 'fig/functype_patched_wrap_fromw32',
'Function Type: <span class="constant">wrap</span> from W32 Code' ]}
<p>This is the simplest case of a function call as it is fully
handled only by the compiler and/or linker.</p>
- <p>In this case though, no debug dumping call relay is provided - such
+ <p>In this case though, no debug dumping call relay is provided — such
relay would need to rename the implementations of native functions to
prevent its automatic linking with the caller code. This renaming would
not be possible to do by simple <span class="constant">#define</span>
<p>The W32 caller which imported the symbol will be pointed right to
the relaying function. The debug dumping relay will be called from W32
code with the appropriate
- <a href="#calltype">cdecl/stdcall/fastcall call type</a> while the
+ @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} while the
relay will call the implementation of the native function in the
- standard UNIX <a href="#calltype_cdecl">cdecl call type</a> manner.</p>
+ standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.</p>
<h4>Native Implementation of "patched" Library Function Called from W32 Code</h4>
</blockquote>
<p>Functions that were not possible to
- <a href="#functype_pass">pass</a> were reimplemented by this project
+ @{[ a_href '#functype_pass','pass' ]} were reimplemented by this project
and placed in the project's implementation directories
- <a href="#reactos_nocare">instead of extending</a> $ReactOS code.</p>
+ @{[ a_href '#reactos_nocare','instead of extending' ]} $ReactOS code.</p>
- <a name="functype_native_wine"><h4>Native Implementation -- <span class="productname">Wine</span></h4></a>
+ <a name="functype_native_wine"><h4>Native Implementation – <span class="productname">Wine</span></h4></a>
<p>Even though $Wine only implements the
<span class="productname">Microsoft Windows NT</span> user space, there
still are some common functions which could be copied from the $Wine
project.</p>
- <a name="functype_native_libcaptive"><h4>Native Implementation - Project Specific</h4></a>
+ <a name="functype_native_libcaptive"><h4>Native Implementation – Project Specific</h4></a>
<p>As the last resort it was necessary to provide completely own
implementation of some API functions such as PC hardware dependent
<a name="calltype"><h2>API Function Calling Conventions</h2></a>
<p>Standard UNIX code compiled by GCC (GNU C Compiler) running on host
- $gnulinux always uses <a href="#calltype_cdecl">cdecl</a> ABI (Application
+ $gnulinux always uses @{[ a_href '#calltype_cdecl','cdecl' ]} ABI (Application
Binary Interface) calling convention. This calling convention is also the
default declaration type of UNIX functions.</p>
arguments are cleaned by the callee. Possible inconsistencies in the
number of function arguments with the function prototype used by the
caller will result in fatal crash. Variable arguments lists cannot be
- passed by this convention - use <a href="#calltype_cdecl">cdecl</a>
+ passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
instead.</p>
<table border="1" align="center">
<span class="constant">EDX</span> respectively. Possible inconsistencies
in the number of function arguments with the function prototype used by
the caller will result in fatal crash. Variable arguments lists cannot be
- passed by this convention - use <a href="#calltype_cdecl">cdecl</a>
+ passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
instead.</p>
<p>GCC (GNU C Compiler) native support for this calling convention
GCC passes them in registers <span class="constant">EAX</span>,
<span class="constant">EDX</span>, <span class="constant">ECX</span>.
This incompatibility is compensated at C source level in the
- <a href="#functype">relaying code</a> generated by
+ @{[ a_href '#functype','relaying code' ]} generated by
<span class="fname">captivesym</span> relay generator.</p>
@{[ doc_img 'fig/calltype_fastcall',
forced <span class="constant">TRUE</span> result of
<span class="function">IoIsOperationSynchronous()</span>
etc.).
- <span class="constant">STATUS_PENDING</span> result code indicating that
- request should be completed in the next callback of the driver is
- considered <a href="#paranoia">fatal</a> as it should not happen for the
- requested synchronous <span class="constant">IRP</span>s (I/O Request
- Packets). Since there is a possibility some filesystem would require
- a real W32 parallel thread all the code that would be hit by W32
+ For several cases needed only by <span class="fname">ntfs.sys</span> there
+ had to be supported asynchronous access
+ (<span class="constant">STATUS_PENDING</span> return code) – parallel
+ execution is emulated by GLib
+ <span class="function">g_idle_add_full()</span> with
+ <span class="function">g_main_context_iteration()</span> called during
+ <span class="function">KeWaitForSingleObject()</span>.</p>
+ Since there is a possibility a real W32 parallel threading would
+ be yet needed in the future all the code that would be hit by W32
multithreading capability is marked by
- <span class="constant">TODO:thread</span> comment for a possible
- future extension.</p>
+ <span class="constant">TODO:thread</span> comment.</p>
<p>Multiple processors (SMP) support will never need to be implemented
since uniprocessor W32 kernels apparently run the filesystem driver modules
<span class="constant">G_DISABLE_ASSERT</span> and
<span class="constant">G_DISABLE_CHECKS</span>.
<span class="productname">Microsoft</span> also produces two versions of
- its products - regular customers use the "free build" (also
+ its products – regular customers use the "free build" (also
called "retail") while the programmers should develop their code
on the "checked build" product releases.</p>
<p>This project has strict consistency checks across all the code to make
the debugging phase easy enough. Failed sanity check is not always
- a bug - sometimes it just means the real W32 binary code is more
+ a bug – sometimes it just means the real W32 binary code is more
benevolent than it could be expected according to the documentation and
such sanity check gets removed for the next version build. In other cases
the failed sanity checks mean the execution path for some unexpected
arguments combination was not yet implemented by this project. I may also
mean a bug, of course...</p>
- <p>Last but not least - never miss a possible sanity check as its
+ <p>Last but not least – never miss a possible sanity check as its
later removal is in an order of magnitude cheaper than an uncaught
invalid assumption. Failed assertion is not always a bug although it
has to be fixed, of course.</p>
- <a name="client_interface"><h2>Client Filesystem Interface</h2></a>
- <p>While this project successfuly communicates with the W32 filesystem
- driver (considered as the lower layer) it must also somehow offer its open
- filesystem interface service to some real client software (upper layer).
- This project offers its own custom filesystem operations interface of <span
- class="constant">libcaptive</span> library based on GLib
- <span class="constant">GObject</span> OO system. Interface prototypes are
- specified in the project's <span class="fname">client-*.h</span>
- include files.</p>
+ <h2>STATUS_LOG_FILE_FULL</h2>
- <p>The filesystem service can be offered in several ways:</p>
+ <p>After writing approx. 1MB of data on NTFS test partition NTFS driver
+ returns for any further write requests
+ <span class="constant">STATUS_LOG_FILE_FULL</status> error code.
+ Apparently it is caused by the fact this project is
+ @{[ a_href '#synchronous','single-threaded' ]} and it ignores the spawn
+ of parallel journalling thread during <span class="fname">ntfs.sys</span>
+ initialization.</p>
- <ul>
- <li>
- <p>One possibility would be to write
- <a name="client_interface_customapp">a custom client application</a>
- for this project such as file manager or a shell. Although it
- would implement the most appropriate user interface to the set of
- functions offered by this project (and W32 filesystem API) it has the
- disadvantage of special client software. Appropriate client is provided
- by this project as:
- <span class="fname">src/client/cmdline/cmdline-captive</span></p>
- </li>
- <li>
- <p>The real UNIX OS filesystem implementation must be completely
- implemented inside the hosting OS kernel. This requires special coding
- methods with limited availability of coding features and libraries.
- Also it would give the full system control to the untrusted W32
- filesystem driver code with possibly fatal consequences of yet
- unhandled W32 emulation code paths. It would benefit from the best
- execution performance but this solution was never considered a real
- possibility.</p>
- </li>
- <li>
- <p>The common approach
- <a name="offered_NFS">of filesystem implementations</a>
- outside UNIX OS kernel were custom NFS servers usually running on the
- same machine as the NFS-connected client as such NFS server is usually
- an ordinary UNIX user space process. It would be possible to implement
- this project as a custom NFS server but the NFS protocol itself
- has a lot of fundamental flaws and complicated code for backward
- compatibility.</p>
- </li>
- <li>
- <p>Currently there is already implemented
- <a name="offered_gnomevfs"><a href="#offered_gnomevfs_todo">Gnome-VFS interface</a></a>
- to the custom filesystem interface of this project's library <span
- class="constant">libcaptive</span>.
- The $GnomeVFSmodule can be used by a Gnome-VFS aware client (such
- as <span class="fname">gnome-vfs/tests/test-shell</span>).</p>
-
- <FIXME:lufs-gvfs>
- <p>The <span class="productname">Gnome-VFS-module</span> can be further
- utilized by the <span class="productname">UserVFS</span>
- \bookcitation{UserVFS-2.0} software ported to provide local <span
- class="productname">Coda</span> \bookcitation{Coda} network filesystem
- server implementation similar to the <a href="#offered_NFS">NFS
- server</a> solution but with much more acceptable network protocol ---
- more about this actual scheme can be found in \link{architecture}{the
- project architecture description}.</p>
- </li>
- <li>
- <FIXME:LUFS>
- <p>Direct interface for the Host-OS kernel would be provided
- by the
- \label{fuse_interface}
- <span class="productname">FUSE</span> \bookcitation{FUSE} project \link{offered_FUSE}{described
- later in this document}. This interface is currently not yet implemented.
- Although it would be much more straightforward than
- <a href="#offered_gnomevfs">Gnome-VFS interface</a> described above,
- its biggest disadvantage would be the requirement to replace/update
- the stock distributions kernel package as it usually does not
- have the <span class="productname">FUSE</span> \bookcitation{FUSE} filesystem support while it already supports
- the <span class="productname">Coda</span> \bookcitation{Coda} interface, which is sufficient for the
- ported <span class="productname">UserVFS</span> \bookcitation{UserVFS-2.0} interface.</p>
- </li>
- </ul>
+ <p>Fortunately <span class="fname">ntfs.sys</span> will clear its
+ journalling log file during filesystem unmount. This project will therefore
+ remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</status>
+ is detected to workaround missing journalling thread.</p>
+
+ <p>Similiar behaviour can be seen during write of compressed files —
+ the file gets written uncompressed and its compression will proceed only
+ during the final filesystem unmount.</p>
- <h2>3rd Party Projects Bugfixes</h2>
+ <p>For these reasons it was mandatory to support
+ @{[ a_href '#parent_connector','transparent volume remounting' ]}.</p>
- <p>Implementation of this project required certain bugfixes to 3rd party
- software packages:</p>
- <h3>GNU Libtool, A Generic Library Support Script</h3>
+ <a name="parent_connector"><h2><span class="constant">ParentConnector</span> volume remounter</h2></a>
+
+ <p>The sandbox master component of this project has control of restarting
+ its sandbox slaves containing the W32 filesystem. Target goal of
+ <span class="constant">ParentConnector</span> component is to transparently
+ provide persistent view of files and directories over the sandboxed slaves
+ being restarted.</p>
- <p><span class="productname"><a href="http://www.gnu.org/software/libtool/">libtool</a></span>:
- Handle duplicate object file names when performing piecewise archive
- linking by renaming object files when needed.</p>
-
- <h3>dosfstools, MS-DOS FAT Filesystems Support on Linux</h3>
-
- <p><span class="productname"><a href="ftp://ftp.uni-erlangen.de/pub/Linux/LOCAL/dosfstools/">dosfstools</a></span>:
- Prevent generation of <span class="constant">FAT-32</span> filesystems
- not supported by the (buggy?) W32 platform
- <span class="fname">fastfat.sys</span> implementation.</p>
-
- <h3>ext2fsd, Ext2 File System Driver</h3>
-
- <p><span class="productname"><a href="http://sys.xiloo.com/projects/projects.htm#ext2fsd">Ext2fsd</a></span>:
- Many filesystem corruption fixes, missing filesystem unregistration
- etc.</p>
-
-
-<h1>Futher Development</h1>
-
- <p>All the W32 filesystem operations of <span class="fname">cdfs.sys</span>,
- <span class="fname">fastfat.sys</span>
- and
- <span class="fname">ext2fsd.sys</span> can be successfuly executed.
- The further development tasks include:</p>
-
- <ul>
- <li>
- <p>The primary goal is to reach <span class="productname">NTFS</span>
- filesystem (<span class="fname">ntfs.sys</span>) compatibility.
- A lot of imported symbols is missing although it is expected most of
- them can be just safely passed for execution in the original
- <span class="fname">ntoskrnl.exe</span>.</p>
- </li>
- <li>
- <p>There may still be valid code paths where some emulated W32 kernel
- functionality and symbols remain unimplemented as these code paths were
- just not hit during testing. The proper way would be to check all the
- possibilities of such code paths execution from the filesystem driver
- code disassembly.</p>
- </li>
- <li>
- <p>No unusual error codes are expected from the filesystem drivers and
- any such return codes will abort the project's execution. For example
- code <span class="constant">STATUS_NO_SUCH_FILE</span> is expected and
- correctly recognized but
- <span class="constant">STATUS_FILE_CORRUPT_ERROR</span> will stop driver
- execution.</p>
-
- <p><a name="exception_fatal">No exceptions in W32 code are allowed</a>
- - any thrown exception will result in driver execution abortion (instead
- of just returning some error code as in the original W32 environment).</p>
-
- <p>These issues should cease to be a problem after deployment of
- sandbox wrapper which will restart the filesystem driver after any
- unexpected error.</p>
- </li>
- <li>
- <p><a name="todo_sandbox">Completion and activation of the sandbox
- wrapper.</a> <span class="fname">src/libcaptive/sandbox/</span> sources
- currently implement the base of both the client and the server sides of
- CORBA interface to separate the client calling filesystem operations from
- the W32 filesystem driver itself. Although CORBA usually makes sense for
- crossmachine network interconnections here it gets a role of
- inter-process interface between the regular client process and the
- <span class="constant">chroot</span>ed/unprivileged/<span class="constant">ulimit</span>ed
- environment of the W32 emulation address space.</p>
-
- <p>Any W32 binary file must be always considered untrusted and therefore
- it is needed to be sandboxed and accessible only via the CORBA interface.
- Furthermore it is needed for clean implementation of $GnomeVFSmodule as
- this project always handles <a href="#mounted_one">exactly one mounted
- filesystem</a> but $GnomeVFSmodule interface expects unlimited number of
- mounts in the scope of one process.</p>
- </li>
- <li>
- <p>Project offers
- <a name="offered_gnomevfs_todo">the filesystem access as its custom UNIX API</a>
- (<span class="fname">captive/client-*.h</span>). This API is currently
- offered in the scope of $GnomeVFSmodule interface as a filter applied to
- the filesystem device (or filesystem image file).
- As $GnomeVFS has no officially supported method of generic $gnulinux
- kernel filesystem access it may be better to provide
- <FIXME:LUFS><a name="offered_FUSE">an interface</a> for <span
- class="productname">FUSE</span> \bookcitation{FUSE} instead.</p>
-
- <p>To get transparent access to W32 filesystems from legacy
- (=non <span class="productname">Gnome-VFS-2.0</span> aware) applications it is possible to use a draft
- port \bookcitation{UserVFS-2.0} of the original <span class="productname">UserVFS</span>
- \bookcitation{UserVFS} to <span class="productname">Gnome-VFS-2.0</span> interface.
- It is also possible to use the test utilities of <span class="productname">Gnome-VFS-2.0</span> \bookcitation{GnomeVFS} package.</p>
- </li>
- <li>
- <p>Implementation of interface to this project by
- <span class="productname"><a href="http://surprise.sourceforge.net/">Partition Surprise</a></span>
- partition manager. Although there currently exists
- <span class="productname"><a href="http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html">ntfsresize</a></span>
- it is a data structures reverse engineered solution which may have
- problems on various hard drives. <span class="productname">Partition
- Surprise</span> project would be able to resize the disk safely by using
- just the original W32 filesystem driver file although with some
- performance hit.</p>
- </li>
- </ul>
+ <p>In the case of read-only operations it would be simple as we could only
+ save our state of currently opened filesystem objects with their read
+ file/directory offset. Write operations can be handled as the read-only
+ ones as long as all the operations are successful. In the case of W32
+ filesystem crash we loose all the past write operations. If we would redo
+ all the write operations we could very easily invoke the same crash.
+ Therefore we write:</p>
+
+ <blockquote class="command">
+ <p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
+ </blockquote>
+
+ <p>message to syslog and refuse any further operations with this
+ object.</p>
+
+ @{[ doc_img 'dia/parent-connector','Parent Connector' ]}
+
+ <p><span class="constant">HANDLE</span> represents W32 object open in
+ existing W32 filesystem.<span class="constant">HANDLE</span> is created
+ on-demand according to the saved state of the object (such as its
+ pathname). Even the whole <span class="constant">VFS</span> sandbox slave
+ is spawn on-demand if some object operation requests it.</p>
+
+ <p>W32 filesystem crash can obviously occur at any moment - it generates
+ @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
+ @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
+ <span class="constant">abort</span>. Successful filesystem unmount
+ (even as the part of remount operation) must be first preceded by
+ <span class="constant">detach</span> signal to close all existing
+ W32 <span class="constant">HANDLE</span>s. After their close the filesystem
+ gets the unmount requests. Only in the case all the close operations
+ succeeded including the final filesystem unmount the signal
+ <span class="constant">cease</span> can be activated to notify all the
+ dirty (written) objects they are now clean. During this
+ <span class="constant">cease</span> signal the project will also
+ @{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
+ underlying media.</p>
+
+ <p>Objects never written remain in <span class="constant">clean</span>
+ state and they can be transparently reopened even if W32 filesystem crash
+ occurs.</p>
+
+
+<h1>TODO: Fsck of NTFS</h1>
+
+ <p>Currently this project does not support checking of data structures
+ of NTFS volume as being provided by <span class="command">chkdsk.exe</span>
+ in W32 environment and <span class="command">fsck</span> in UNIX OS.</p>
+
+ <p>W32 has its disk checking functionality split to
+ <span class="fname">untfs.dll</span> W32 userland library.
+ according to
+ @{[ a_href 'http://www.sysinternals.com/ntw2k/source/fmifs.shtml',
+ 'Chkdskx and Formatx' ]}
+ by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
+ 'Mark Russinovich' ]}.
+
+ <p>I assume its execution falls completely
+ @{[ a_href '#existing_emulation','out of scope' ]}
+ of this project as it is W32 userland.</p>
+
+ <p>This possibility was not yet investigated in any way.</p>
+
+
+<h1>TODO: NTFS Support for
+ <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span></h1>
+
+ <p>Although there currently exists
+ <span class="productname">@{[ a_href 'http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html','ntfsresize' ]}</span>
+ I am not sure whether it is really reliable for all NTFS filesystems.
+ <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span>
+ is the only partition manager capable of safely resize the disk
+ by using just the original W32 filesystem driver by full rebuild of
+ filesystem metadata.
+ Almost no file data blocks would be moved even on these generic filesystems
+ as W32 supports <span class="constant">FSCTL_MOVE_FILE</span> request
+ according to
+ @{[ a_href 'http://www.sysinternals.com/ntw2k/info/defrag.shtml',
+ 'Inside Windows NT Disk Defragmenting' ]}
+ by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
+ 'Mark Russinovich' ]}.
<h1>Related Projects</h1>
<p>Although this project takes a completely different approach and has
a different architecture, the final goal is the same as for this
- project - reliable read-write <span class="productname">NTFS</span>
+ project – reliable read-write <span class="productname">NTFS</span>
filesystem support. $LinuxNTFS goes the way of reverse engineering
filesystem data structures (and possibly
<span class="fname">ntfs.sys</span> itself). Unfortunately after many years
<span class="fname">ntfs.sys</span>) files from the user's
<span class="productname">NTFS</span> partition.</p>
- <h2><span class="productname"><a href="http://www.cgsecurity.org/ntfs.html">NTPwd NTFS Driver</a></span></h2>
+ <h2><span class="productname">@{[ a_href 'http://www.cgsecurity.org/ntfs.html','NTPwd NTFS Driver' ]}</span></h2>
- <p>DOS based <a href="http://www.gnu.org/licenses/gpl.html">GPL-2.0</a>
+ <p>DOS based @{[ a_href 'http://www.gnu.org/licenses/gpl.html','GPL-2.0' ]}
read-write NTFS driver. Filesystem structures are reverse engineered in the
- way of <a href="#LinuxNTFScompet">Linux-NTFS Project</a>. As it is not very
+ way of @{[ a_href '#LinuxNTFScompet','Linux-NTFS Project' ]}. As it is not very
actively maintained it reaches a lower level of
<span class="productname">NTFS</span> compatibility.</p>
- <h2>Virtual Machine with <span class="productname">Microsoft Windows NT</span></h2>
+ <h2>The only real competition: Closed-source read/write @{[ '$299' ]} equivalent</h2>
+
+ <p>@{[ a_href 'http://www.vmware.com/download/workstation.html',
+ 'VMware Workstation' ]}</p>
+
+ <p>Original Microsoft Windows operating system can be run inside a virtual
+ machine running under GNU/Linux and share the read-write NTFS disk by using
+ a network file sharing through a VMware virtual network card.</p>
+
+ <p>You need @{[ '$299' ]} for this product and you need to
+ give up your system security by running un@{[ a_href '#sandbox','sandbox' ]}ed
+ closed-source program in your GNU/Linux.</p>
+
+ <h2>@{[ a_href 'http://www.winehq.com/','Wine Project' ]}</h2>
+
+ <p>No code could be shared – Wine emulates only Microsoft Windows userland.
+ Filesystem drivers completely belong to Microsoft Windows kernelland.</p>
+
+ <h2>@{[ a_href 'http://www.sysinternals.com/ntw2k/freeware/ntfswin98.shtml','NTFS for Windows 98' ]}</h2>
+
+ <p>Closed-source read-only-crippled @{[ '$0' ]} equivalent for Microsoft Windows.</p>
+
+ <p>There is a @{[ a_href 'http://www.sysinternals.com/images/screenshots/ntfs98ap.gif',
+ 'diagram' ]} showing exactly the principle of Captive NTFS project.
+ There is apparently disabled read/write functionality in <i>NTFS for
+ Windows 98</i> as the same company also sells the following product sharing
+ the same codebase:</p>
+
+ <h2>@{[ a_href 'http://www.winternals.com/products/repairandrecovery/ntfsdospro.asp','NTFSDOS Professional' ]}</h2>
+
+ <p>Closed-source read/write @{[ '$299' ]} equivalent for MS-DOS.</p>
+
+ <p>This product is the most close equivalent to Captive NTFS but it is
+ a commercial product, closed-source and it has filesystem interface only
+ for MS-DOS.</p>
+
+
+<h1>Re: @{[ a_href 'http://linux-ntfs.sourceforge.net/info/ntfs.html#7.7',
+ "7.7 Can't we write a wrapper for Windows' driver?" ]}</h1>
+
+ <p class="re">> It sounds like a great idea, to start with, but there are numerous
+ problems.</p>
+
+ <p><span class="re">> The largest technical problem is joining the Windows
+ system DLL to the Linux VFS. It could be done, but it wouldn't be pretty.</span><br />
+ Yep. :-)</p>
+
+ <p><span class="re">> It would have to run as part of the kernel which would mean
+ that if it went wrong it could crash the machine. With no source, we might not
+ be able to work around the problem.</span><br />
+ @{[ a_href '#sandbox','Nope' ]},
+ @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
+ moves the filesystem implementation to UNIX userland where the Microsoft
+ Windows filesystem is completely unarmed by Captive jail of chroot(2),
+ setuid(2) and setrlimit(2). There only remains one narrow connection to the rest of
+ system (by CORBA/ORBit). The filesystem's life environment gets kill(2)ed when
+ UNIX is no longer satisfied with it. Safety similiar to
+ @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware sandbox' ]}.</p>
+
+ <p><span class="re">> The next major problem is compati<!--orig. text typo-->bility.
+ Which version of the Windows system file would we use? Picking one would limit
+ its use, making the wrapper versatile for all of them would be a programming
+ nightmare.</span><br />
+ Microsoft Windows NTFS filesystem driver is capable of accessing older formats
+ of the filesystem. This project currently runs Microsoft Windows XP version,
+ porting to Microsoft Windows 2003 Server expected. (Microsoft Windows upgrades
+ NTFS disk filesystem to its own version during complete CD-ROM Microsoft
+ Windows system installation – such operation is not threat this project use.)</p>
+
+ <p><span class="re">> And it gets worse. The legal implications of
+ distributing Windows systems files would cause problems.</span><br />
+ User must be careful to obey all licensing restrictions according to his
+ local country laws.<br />
+ <span class="re">> Also the proprietary nature of the driver would mean that
+ the other kernel coders would not investigate any problems if someone had used
+ the NTFS wrapper.</span><br />
+ It does not apply to this project due to the implemented
+ @{[ a_href '#sandbox','filesystem separation' ]}.</p>
+
- <p>Original <span class="productname">Microsoft Windows NT</span>
- operating system can be run inside a virtual machine running under
- $gnulinux (or vice versa) and share the read-write disk partitions by using
- a network file sharing through a virtual network card.</p>
-
- <p>Although there will be full filesystem structures compatibility the
- <span class="productname">NTFS</span> partition cannot be accessed with no
- system installed (or with non-bootable crashed system to repair it this
- way)
- (Although this project requires the original
- <span class="fname">ntfs.sys</span> it can obtained from the legal
- <span class="productname">Microsoft Windows NT</span> CD.),
- it will have substantial system resources requirement and you also need
- a virtual machine software product such as commercial
- <span class="productname"><a href="http://www.vmware.com/download/workstation.html">VMware Workstation</a></span>.</p>
-
-
-<h1>Conclusion</h1>
-
- <p>The project established <a href="#existing_emulation">a new form</a>
- of W32 emulation model suitable for existing proprietary binary W32 kernel
- code (drivers) while being hosted in an open source operating system
- (currently $gnulinux). Currently, only the subsystems required by W32
- filesystem drivers are implemented but the project can be further extended
- for compatibility with various hardware-related drivers such as W32 video
- drivers, W32 disk interface drivers etc.</p>
-
- <p>Some W32 kernel space subsystems were implemented for the first time as
- $freespeech code as they are still missing in the only currently available
- $freespeech W32 kernel implementation, $ReactOS. Some W32 kernel function
- behaviour expected by the drivers had to be reverse engineered and documented
- in this project's API documentation (not listed in this book) and/or in
- its source files, because its description in the
- <span class="productname">Microsoft</span> documentation is missing.</p>
-
- <p>Author had to get familiar both with the W32 kernel API and also with the
- W32 kernel code by the reverse engineering. This experience also covers the
- first <span class="productname">Microsoft Windows</span> compatible code ever
- written by the author - <span class="fname">hal.dll</span> (Hardware
- Abstraction Layer) part of W32 kernel.</p>
-
- <p>Certain UNIX implementation interfaces allow a regular, non-privileged
- user of UNIX system to mount image files with any W32 filesystem supported by
- this project. Such mount operation usually requires UNIX
- <span class="constant">root</span> privileges to do so. On the other hand the
- choice of supported filesystem types is very limited as only a few
- filesystem types are supported for the W32 platform.</p>
HERE
git://git.jankratochvil.net / www.jankratochvil.net.git / commitdiff
