[www.jankratochvil.net.git] / project / captive / doc / CacheManager.html.pl

#! /usr/bin/perl
# 
# $Id$
# Captive project doc Cache Manager page Perl template.
# Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; exactly version 2 of June 1991 is required
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA


package project::captive::doc::CacheManager;
require 5.6.0;  # at least 'use warnings;' but we need some 5.6.0+ modules anyway
our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
our $CVS_ID=q$Id$;
use strict;
use warnings;

BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
use My::Web;
require CGI;
BEGIN { Wuse 'project::captive::doc::Macros'; }


project::captive::doc::Macros->init(
                "__PACKAGE__"=>__PACKAGE__,
                "title"=>'Captive NTFS Developer Documentation: NT Cache Manager',
                "rel_prev"=>'Reverse.html.pl',
                "rel_next"=>'Details.html.pl',
                );


print <<"HERE";


<a name="cache_manager"><h1>NT Cache Manager</h1></a>

        <p>Although there exist some 3rd party documents about
        <span class="productname">NT Cache Manager</span> W32 subsystem such as
        <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
                        'The NT Cache Manager Description' ]}</span> or
        <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
                        'Learn About NT'."'".'s&nbsp;File-system Cache' ]}</span>
        they are definitely insufficient for compatible
        <span class="productname">NT Cache Manager</span> reimplementation.</p>

        <p><span class="productname">NT Cache Manager</span> is about mapping
        filesystem objects such as regular file data, filesystem bitmap or
        journalling zone (log file). It is also being used by the filesystem for
        mapping of virtual volume files representing the whole underlying
        filesystem device.</p>

        <p>The original W32 <span class="productname">NT Cache Manager</span>
        is much more complicated as it must coordinate its effort with
        other W32 subsystems like mapping of executable files
        (<span class="type">ImageSectionObject</span>), insufficient system
        resources from <span class="productname">NT Memory Manager</span>
        or general effort to perform caching features for system performance.</p>
        <span class="productname">NT Cache Manager</span> of this project has much
        simpler goal - it just needs to provide compatible
        <span class="productname">NT Cache Manager</span> functionality while
        the other goals of its W32 counterpart are left to be successfuly handled
        by UNIX OS in much more efficient way.</p>

        @{[ doc_img 'dia/cache-manager',
                        '<span class="productname">NT Cache Manager</span> Architecture' ]}

        <p>Cache Manager objects are always bound to
        <span class="type">FCB</span> (File Control Block).
        <span class="type">FileObject</span> (or its associated
        <span class="type">HANDLE</span>) serve only as reference
        to <span class="type">FCB</span> and there can be multiple
        <span class="type">FileObject</span>/<span class="type">HANDLE</span>
        items for one <span class="type">FCB</span>. It is a bit misleading
        you must use <span class="type">FileObject</span> pointer while calling
        most of the Cache Manager functions.</p>

        <p>Before using any other Cache Manager functions you must first call
        <span class="function">CcInitializeCacheMap()</span>. You must give the
        maximum mapped object offset. Each mapped object byte must have at most one
        mapped memory location - no shared pages are allowed. Also any subsequent
        mapping request is expected to be mapped into continuous memory region.
        It implies you must reserve the memory region for possible future mapping
        during the initial <span class="function">CcInitializeCacheMap()</span>
        moment sized according to the given maximum mapped object offset. 
        This is the approach currently implemented by this project although it
        cannot be used for 3rd party <span class="fname">ext2fsd.sys</span>
        driver as it initialized Cache Manager by the whole media device size
        and it surprisingly succeeds for original
        <span class="productname">Microsoft Windows</span> 
        <span class="productname">Cache Manager</span>.
        I expect the space reservation should be postponed to the first mapping
        request and expect no multiple mappings will be done in the case
        of memory-exceeding <span class="function">CcInitializeCacheMap()</span>
        reservation request. <span class="function">CcSetFileSizes()</span>
        changing the reserved memory area size may assume no existing Map
        or Pin mappings exist. Only in the case of 
        <span class="constant">FO_STREAM_FILE</span> (virtual device file)
        it is permitted to extend mapped size even in the case of existing
        (and dirty) Map or Pin mappings.</p>

        <p><span class="type">PCACHE_MANAGER_CALLBACKS</type> argument can be
        safely ignored:</p>

        <dl>
                <dt><span class="function">AcquireForReadAhead()</span>/<span class="function">ReleaseFromReadAhead()</span></dt>
                <dd>
                        <p>As any readahead functionality is optional these entries are
                        never used by Cache Manager implementation of this project.</p>
                </dd>

                <dt><span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span></dt>
                <dd>
                        <p>Even the write-behind functionality is optional for Cache Manager.
                        It is being done in asynchronous way in the original
                        <span class="productname">Microsoft Windows</span>
                        <span class="productname">Cache Manager</span>.
                        implementation and it is ignored by Cache Manager implementation of
                        this project.</p>

                        <p>Cache Manager does not need to write any data if not explicitely
                        requested by the driver. It is even expected to silently drop any
                        pending dirty data blocks during filesystem shutdown.
                        Forced dirty block write by function
                        <span class="function">CcFlushCache()</span> should be written without
                        any wrapping surrounding
                        <span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span>
                        pair.</p>
                </dd>
        </dl>

        <p><span class="function">CcUninitializeCacheMap()</span> is just
        a suggestion for Cache Manager that driver will no longer reference
        given <span class="type">SharedCacheMap</span>. The uninitialization
        can be postponed to any later moment in original 
        <span class="productname">Microsoft Windows</span> 
        <span class="productname">Cache Manager</span>
        as it may be locked by existing
        <span class="type">ImageSectionObject</span>
        of some file being executed etc.
        <a name="sharedcachemap_leak">It is fatal to destroy
        <span class="type">SharedCacheMap</span></a>
        in the moment you see no other
        references to it as the driver will access it for some moment
        even after <span class="function">CcUninitializeCacheMap()</span>.
        I am not sure if it is a bug of the driver or whether there are some rules
        how long after <span class="function">CcUninitializeCacheMap()</span>
        completion given <span class="type">SharedCacheMap</span> still exists.
        Fortunately it is safe to never destroy
        <span class="type">SharedCacheMap</span> and leave it leaked - everything
        gets clean in the
        @{[ a_href 'Details.html.pl#sandbox','sandboxed environment' ]} soon anyway.</p>

        <p>There exist Map and Pin type objects for each
        <span class="type">SharedCacheMap</span> although they look very similiar.
        Only these objects give you access to any memory data
        &mdash; <span class="type">SharedCacheMap</span> only reserved the space
        to ensure continuous mapping of the forthcoming mappings but it did not map
        any data into it.</p>

        <p>Mapping of 'new' Map or Pin will create the new object only in the case
        no such mapping exists now. Otherwise you will just get the reference to
        the existing object with increased usecount.</p>

        <dl>
                <dt>Map</dt>
                <dd>
                        <p>Map mapping is always at most one for each
                        <span class="type">SharedCacheMap</span>. Base offset/length of such
                        mapping have no meaning as there can be only single Map.</p>

                        <p>Apparently Map size can be arbitrary long according
                        to its <span class="type">SharedCacheMap</span> reserved space.</p>

                        <p>You cannot modify the memory mapped by Map in any way.
                        As it is the same memory area (address) as the pages used by Pin
                        objects you always access the last modified version by possible
                        Pin of the same page.</p>
                </dd>

                <dt>Pin</dt>
                <dd>
                        <p>Pin mapping always represents just one physical page
                        (<span class="constant">PAGE_SIZE</span> &nspan; 4096 for i386).
                        Its base offset/length can be safely extended to be aligned to the
                        requested page.</p>

                        <p>Pin can have associated pair of oldest and newest
                        <span class="type>LSN</span> (Linear Sequence Number). It can be
                        set by <span class="function">CcSetDirtyPinnedData()</span>
                        and Cache Manager always tracks the lowest and highest
                        reported <span class="type>LSN</span> for each page.
                        <span class="type>LSN</span> is assumed to be
                        <span class="constant">0</span> if not set.</p>

                        <p>Any existing Pin mapping will be reused for further mappings
                        as long as it is not ThreadOwned. In the moment you use
                        <span class="function">CcSetBcbOwnerPointer()</span> you will detach
                        the associated Pin pages from its
                        <span class="type">SharedCacheMap</span>.
                        Although they will further act as valid Pin mappings they will be no
                        longer reused during new Pin mapping of the same page.
                        There can exist multiple Pin mappings of the same page (although
                        sharing the same memory space). This detaching must be implemented
                        even in the
                        @{[ a_href 'Details.html.pl#synchronous','single-threaded' ]} W32 implementation
                        of this project as it is affecting the behaviour of Cache Manager.
                        It was never
                        @{[ a_href 'CacheManager.html.pl#TraceFS','seen' ]} how to behave if multiple dirty Pin
                        mappings of the same page exist.</p>
                </dd>
        </dl>

        <p>Only the pages not yet present in the memory must be read from the disk.
        You must not read any pages you do not need to as the driver does not
        expect it and it would corrupt its data buffers. There is just a&nbsp;strict
        difference between <span class="function">CcPinRead()</span> and
        <span class="function">CcPinMappedData()</span> function calls where
        <span class="function">CcPinRead()</span> is required to re-read its data
        blocks even if they were currently already Map mapped (unless it was already
        also Pin mapped at least once). On the opposite side
        <span class="function">CcPinMappedData()</span> must not re-read the given
        blocks, moreover it blocks are required to be already Map mapped by the caller.</p>

        <p>Cache Manager of this project will destroy Pin or Map mappings after
        their last unreferencing (in opposite of
        @{[ a_href 'sharedcachemap_leak','leaked <span class="type">SharedCacheMap</span>' ]}).
        Despite it any dirty pages may still be held as the pages
        (including their <span class="type>LSN</span>s) are cached associated
        with <span class="type">SharedCacheMap</span>. It may be also possible
        original <span class="productname">Microsoft Windows</span> 
        <span class="productname">Cache Manager</span>
        postpones Pin mapping destroy to later time but it does not matter.</p>


        <a name="TraceFS"><h2>TraceFS NT Cache Manager Tracer</h2></a>

                <p>@{[ a_href '#cache_manager','Cache Manager behaviour' ]} would be hard
                to analyze just by @{[ a_href '#reverse','reverse engineering' ]} as it
                is pretty complicated code cooperating with many other W32 kernel
                subsystems. It was chosen as easier way to trace it instead and validate
                all the Cache Manager assumptions by Cache Manager simulator.</p>

                @{[ doc_img 'dia/TraceFS','TraceFS Hooking' ]}

                <p>You must prepare your driver to be hooked
                (<span class="fname">ntfs.sys</span> in this case):</p>

                <blockquote class="command">
                        <p>@{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} ntfs.sys ./src/TraceFS/TraceFS-W32/TraceFS.sys &gt;hooked/ntfs.sys</p>
                </blockquote>

                <p>This <span class="fname">hooked/ntfs.sys</span> file must be replaced
                in the <span class="fname">%System32%\\drivers</span> directory.
                Beware as
                <span class="productname">Microsoft Windows</span>
                has many backups of these system files such as
                <span class="fname">%System32%\\dllcache</span> &mdash; delete them
                all!</p>

                <p>You also need to install
                <span class="fname">./src/TraceFS/TraceFS-W32/TraceFS.sys</span>
                into <span class="fname">%System32%\\drivers</span> directory
                and import <span class="fname">TraceFS/TraceFS-W32/TraceFS.reg</span>
                registry file to initialize the debug driver during system boot.</p>

                <p>You can now pray a bit and snap the resulting Cache Manager tracing
                from <span class="productname">WinDbg</span> by
                @{[ a_href 'Reverse.html.pl#WinDbg','W32 remote kernel debugging' ]}:</p>

                @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}

                <p>The resulting trace file should be processed by
                @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
                implementation to validate its assumptions about Cache Manager behaviour.
                Any seen incompatibilies will be reported &mdash; your target is to reach
                as few error messages as possible.</p>

                <p>KNOWN BUGS: Combination of message synchronization primitives and
                implemented refusal to create journalling thread of
                <span class="fname">ntfs.sys</span>
                causes fatal system lockup in several advanced operations
                such as setting compression attribute. Despite it more common operations
                can be successfuly traced during the whole
                <span class="productname">Microsoft Windows</span>
                session including its final shutdown and such traces provide enough
                material to be food to
                @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
                validator.</p>

                <a name="TraceFS_general"><h3>TraceFS for general API tracing</h3></a>

                        <p>Although TraceFS was up to now used only for tracing of
                        <span class="productname">NT Cache Manager</span> it can be easily
                        used ever for any other NT kernel API tracing. You need to provide
                        appropriate function wrappers in the main source file
                        @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.c' ]}.
                        Original system functions being wrapped should be called with their
                        original name. Your wrapping functions should have the first letter
                        of their name replaced by character
                        <span class="command">'T'</span> - wrapping of
                        <span class="function">CcInitializeCacheMap()</span> must be
                        done be your function
                        <span class="function">TcInitializeCacheMap()</span>.
                        Prototypes of both the wrapping and wrapped functions must be the same.
                        You must also export all the wrapped functions by
                        @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.def' ]}.
                        @{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} has no hardcoded
                        function names &ndash; it will hook exactly the exported entries.</p>

                        <p>Framework for thread synchronizations and debug tracing is provided to
                        prevent mangling of messages while running by multiple threads at once.
                        Testing was done just on uniprocessor machine, SMP kernel may need some
                        fixes.</p>
                

HERE


project::captive::doc::Macros->footer();