2 # Captive project doc Components page Perl template.
3 # Copyright (C) 2003-2005 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
5 # This program is free software; you can redistribute it and/or modify
6 # it under the terms of the GNU General Public License as published by
7 # the Free Software Foundation; exactly version 2 of June 1991 is required
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
14 # You should have received a copy of the GNU General Public License
15 # along with this program; if not, write to the Free Software
16 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
19 package project::captive::doc::Components;
20 require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
21 our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
31 BEGIN { Wuse 'project::captive::doc::Macros'; }
32 project::captive::doc::Macros->init(
33 "__PACKAGE__"=>__PACKAGE__,
34 "title"=>'Captive NTFS Developer Documentation: Components',
35 "rel_prev"=>'Architecture.pm',
36 "rel_next"=>'Reverse.pm',
43 <h1>Project Components</h1>
45 @{[ doc_img 'dia/arch-all.dia','Project Components Architecture' ]}
47 <p>Most of the work of this project is located in the single box called
48 "<span class="constant">libcaptive</span>" located in the center
49 of the scheme. This component implements the core W32 kernel API by
50 various methods described in this document.
51 The "<span class="constant">libcaptive</span>" box cannot be
52 further dissected as it is just an implementation of a set of
53 @{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym','API functions' ]}.
54 It could be separated to several subsystems such as the
55 @{[ a_href '#cache_manager','Cache Manager' ]},
56 Memory Manager, Object Manager, Runtime Library, I/O Manager
57 etc. but they have no interesting referencing structure.</p>
59 <p>As this project is in fact just a filesystem implementation every
60 story must begin at the device file and end at the filesystem operations
61 interface. The unified suppported interfaces are
62 <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
63 (the most low level portability, data-types and utility library for Gnome)
64 <span class="type">GIOChannel</span> (for the device access) and the custom
65 <span class="constant">libcaptive</span> filesystem API. Each of these ends
66 can be connected either to some direct interface (such as the
67 <span class="constant">captive-cmdline</span> client),
68 @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
69 or as a general $GnomeVFS filter.
70 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
71 most cases as it offers standard filesystem interface by Linux kernel.
73 You can also use $GnomeVFS as it offers nice filter interface on
74 the UNIX user-privileges level for transparent operation with archives and
75 network protocols. This filter interface was used by this project to turn
76 the device reference such as <span class="fname">/dev/hda3</span> or <span
77 class="fname">/dev/discs/disc0/part3</span> to the fully accessible
78 filesystem (pretending being an "archive" in the device
79 reference). This device access can be specified by $GnomeVFS URLs such as:
81 class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
83 <p><span class="constant">captive-bug-replay</span> serves just for debugging
84 purposes — you can 'replay' existing
85 <span class="fname">file.captivebug.xml.gz</span> automatically being
86 generated during W32 filesystem failure. This bugreport file will contain
87 all the touched data blocks of the device used in the moment of the
88 failure. <span class="constant">captive-bug-replay</span> will therefore
89 emulate internal virtual writable device out of these bugreported data.</p>
91 <p>If the passed device reference is requested by the user to be accessed
92 either in <span class="dashdash">--ro</span> (read-only) mode or in the
93 <span class="dashdash">--rw</span> (full read-write) mode there are no
94 further device layers needed. Just in the case of <span
95 class="dashdash">--blind</span> mode another layer is involved to emulate
96 read-write device on top of the real read-only device by the method of
97 non-persistent memory buffering of all the possible write requests.</p>
99 <p><span class="constant">sandbox commit buffer</span> is involved only in the
100 case @{[ a_href 'Details.pm#sandbox','sandboxing feature' ]} is active. It will
101 buffer any writes to the device during the sandbox run to prevent
102 filesystem damage if the driver would fail in the meantime. If the
103 filesystem gets finally successfully unmounted this sandbox buffer can be
104 <span id="safe_flush">safely flushed</span>
105 to its underlying physical media. The buffer will be dropped
106 in the case of filesystem failure, of course. The filesystem should be
107 unmounted from time to time — it can be transparently unmounted and mounted
108 by <span class="command">commit</span> of
109 <span class="constant">captive-cmdline</span> custom client. Currently you
110 cannot force remounting when using
111 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
112 but it will be remounted after approx each 1MB data written automatically
113 due to @{[ a_href '#log_file_full','NTFS log file full' ]}.
115 Now we need to transparently
116 @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','connect' ]}
117 the device interface of <span class="type">GIOChannel</span> type through
118 @{[ a_href 'Details.pm#sandbox','CORBA/ORBit' ]} to the sandboxed slave.</p>
120 <p>Such device is still only a UNIX style GLib <span
121 class="type">GIOChannel</span> type at this point. As we need to supply it
122 to the W32 filesystem driver we must convert it to the W32 I/O Device
123 with its capability of handling <span class="type">IRP</span>
124 (<span class="constant">I/O Request Packet</span>; structure holding the
125 request and result data for any W32 filesystem or W32 block device
127 requests from its upper W32 filesystem driver. Such W32 I/O Device can
128 represent either <span class="type">CD-ROM</span> or
129 <span class="type">disk</span> device type as different W32 filesystem
130 drivers require different media types — currently only
131 <span class="fname">cdfs.sys</span> requires
132 <span class="type">CD-ROM</span> type.</p>
134 <p>W32 media I/O Device is accessed from the W32 filesystem driver.
135 The filesystem driver itself always creates volume object by
136 <span class="function">IoCreateStreamFileObject()</span> representing the
137 underlying W32 media I/O Device as the object handled by the
138 filesystem driver itself. All the client application filesystem requests
139 must be first resolved at the filesystem structures level, passed to the
140 volume stream object of the same filesystem and then finally passed to the
141 W32 media I/O Device (already implemented by this project as an
142 interface to <span class="type">GIOChannel</span> noted above).</p>
144 <p>The filesystem driver is called by the core W32 kernel implementation of
145 <span class="constant">libcaptive</span> in
146 @{[ a_href 'Details.pm#synchronous','synchronous way' ]} in single-shot manner instead of
147 the several reentrancies while waiting for the disk I/O completions as can
148 be seen in the original
149 <span class="productname">Microsoft Windows NT</span>.
150 This single-shot synchronous behaviour is possible since all the needed
151 resources (disk blocks etc.) can be always presented as instantly ready as
152 their acquirement is solved by @{[ a_href 'Architecture.pm#hostosnote','Host-OS' ]} outside of
153 the W32 emulated @{[ a_href 'Architecture.pm#guestosnote','Guest-OS' ]} environment.
154 For several cases needed only by <span class="fname">ntfs.sys</span>
155 there had to be supported asynchronous access — parallel execution
156 is emulated by GLib <span class="function">g_idle_add_full()</span>
157 with <span class="function">g_main_context_iteration()</span> called during
158 <span class="function">KeWaitForSingleObject()</span>.</p>
160 <p><span class="constant">libcaptive</span> offers the W32 kernel
161 filesystem API to the upper layers. This is still not the API the common
162 W32 applications are used to as they use W32 libraries which in turn pass
163 the call to W32 kernel. For example
164 <span class="function">CreateFileA()</span> is being implemented by several
165 libraries such as <span class="fname">user32.dll</span> as a relay
166 interface for the kernel function
167 <span class="function">IoCreateFile()</span> implemented by this
168 project's <span class="constant">libcaptive</span> W32 kernel
169 emulation component.</p>
171 <p>As it would be very inconvenient to use the legacy, bloated and UNIX
172 style unfriendly W32 kernel filesystem API this project offers its own
173 @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
174 the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
175 This interface is supposed to be easily utilized by
176 <a href="#client_interface_customapp">a custom application accessing
177 the W32 filesystem driver</a>.</p>
179 <p>@{[ a_href 'Details.pm#sandbox','CORBA/ORBit' ]} hits us again – we need to
180 @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','translate' ]}
181 the @{[ a_href '#client_interface','custom filesystem API interface' ]}
182 out of the sandboxed slave to the UNIX space.</p>
184 <p><span class="constant">captive sandbox master</span> provides the
185 functionality of covering any possible sandboxed slave restarts and its
186 communication. It is also capable of
187 <span id="demultiplexing_master">demultiplexing single API operations</span>
188 to multiple its connected sandbox slaves in transparent way
189 as each of them handles
190 @{[ a_href 'Details.pm#mounted_one','just one filesystem device' ]}.</p>
192 <p>The rest of the story is not much special for this project since this is
193 a common UNIX problem how to offer user space implemented UNIX filesystem
194 as a generic system filesystem (as those are usually implemented only as
195 the components od UNIX kernel).</p>
197 <p>The filesystem service can be offered in several ways:</p>
200 <dt>Custom client</dt>
202 <p>One possibility would be to write
203 <span id="client_interface_customapp">a custom client application</span>
204 for this project such as file manager or a shell. Although it
205 would implement the most appropriate user interface to the set of
206 functions offered by this project (and W32 filesystem API) it has the
207 disadvantage of special client software. Appropriate client is provided
209 <span class="fname">src/client/cmdline/cmdline-captive</span></p>
212 <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
214 <p>The most usable interface is the
215 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
216 by <span class="constant">liblufs-captivefs</span>.
217 As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
218 already assigns separate process for each filesystem mount the
219 @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
220 is not utilized in this case.</p>
222 <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
223 needs multiple operating threads (each UNIX kernel operation needs
224 one free lufsd slot/thread to not to fail immediately).
225 As <span class="constant">libcaptive</span> is
226 @{[ a_href 'Details.pm#synchronous','single-threaded' ]} all the operations
227 get always synchronized by
228 <span class="constant">liblufs-captivefs</span>
229 before their pass over to <span class="constant">libcaptive</span>.</p>
232 <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
234 <p>This client allowing its filesystem access even without any
235 involvement of UNIX kernel from any $GnomeVFS aware client application
236 (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
237 This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
238 data flow of this project in two points — both as the lowest layer
239 device image source and also as the upper layer for the filesystem
240 operation requests.</p>
244 <p>Unimplemented and deprecated methods for providing filesystem
248 <dt>W32 filesystem in UNIX OS kernel</dt>
250 <p>The real UNIX OS filesystem implementation must be completely
251 implemented inside the hosting OS kernel. This requires special coding
252 methods with limited availability of coding features and libraries.
253 Also it would give the full system control to the untrusted W32
254 filesystem driver code with possibly fatal consequences of yet
255 unhandled W32 emulation code paths. It would benefit from the best
256 execution performance but this solution was never considered a real
260 <dt>Custom NFS server</dt>
262 <p>The common approach
263 <span id="offered_NFS">of filesystem implementations</span>
264 outside UNIX OS kernel were custom NFS servers usually running on the
265 same machine as the NFS-connected client as such NFS server is usually
266 an ordinary UNIX user space process. It would be possible to implement
267 this project as a custom NFS server but the NFS protocol itself
268 has a lot of fundamental flaws and complicated code for backward