$Id$ 3.41 2019/12/31 (dwd) - When a "X-Frontier-Opts: DontCacheErrors" request header is set, abort error responses before they send the zero-length http/1.1 last chunk. This prevents proxies from caching the response. 3.40 2018/08/20 (dwd) - Ignore any text before the BEGIN line in the private key file. This appears to have been done in 3.33 but lost in 3.34. 3.39 2018/01/23 (dwd) - When responding with an error condition before headers are sent, instead of setting a Last-Modified header to be blank, set it to the oldest possible time of midnight on January 1, 1970. With the old behavior, Squid-3 does not clear the cache even though Squid-2 does. With the new behavior, squid-3 always clears the cache even when another error sends the same Last-Modified time. 3.38 2017/05/24 (dwd) - Add log message after DB data transferred including the msecs it took to transfer the data. 3.37 2017/03/22 (dwd) - Fix bug introduced in 3.36 that caused a null pointer exception in in the FilePlugin. 3.36 2017/02/08 (dwd) - Support HTTP 1.1 chunked encoding to backend FilePlugin http servers. Note that the protocol to the client can now send multiple byte arrays per request instead of just one. An implication is that fn-fileget from frontier-client versions older than 2.8.21 will not always retrieve all of the data when the backend is an http server (although it will when reading directly from files). - Support MySQL as a database backend in the SQLPlugin. This required changing the Last-Modified handling to use the MySQL table that always keeps track of modification times on all tables (it does not require an add-on function like oracle does). 3.35 2016/10/09 (dwd) - Add new servlet option ExpireEmptyQueriesLikeErrors, default false, which when true causes empty queries to be expired at the same shorter age as errors (default 5 minutes). Previously this was always true. - When ExpireEmptyQueriesLikeErrors is true, mark the returned payload with the shorter max_age so if a cached item age exceeds the time, the client will repeat the request with the shorter max age. This can happen because a later refresh with If-Modified-Since when the Last-Modified time hasn't changed, the regular long expiration time will be returned because the query is not repeated. 3.34 2014/10/29 (dwd) - Support host private keys (for digital signatures) that start with 'BEGIN PRIVATE KEY' in addition to 'BEGIN RSA PRIVATE KEY'. That form has begun to be used at CERN with 4096-bit RSA keys; the old form was previously used for 2048-bit keys. - Eliminate an infinite loop when there's an '&' in an error message as sometimes happens with the FilePlugin. - Do one less level of url decoding on the query string put into catalina.out logs, because one level is already done and that better matches the URL seen by a client using the FilePlugin. 3.33 2013/12/26 (dwd) - Added new "forever" time to live, a third cache expiration time requested by clients, in addition to "short" and "long" times to live. Added config.properties option ForeverCacheExpireSeconds which defaults to a year of seconds. If a "forever" query returns empty results, it is considered to be an error. - Reduced the time that empty query results will be cached for "short" or "long" time to live queries to a maximum of 5 minutes or the "short" time, whichever is smaller. - Added a "max_age" value at the end of the response if the http header has already been sent with a longer Cache-control: max-age than is later determined to be necessary. The things that can cause the http header to be sent are either a keepalive message because the request had to wait longer than 5 seconds to read from the database, or at least a buffer full of data was already sent before an error occurred. The events that can reduce the max age late are some kind of error with the database (which used to be handled by only signalling "global" errors to the client) or an empty result from a database query. - Show a more user friendly error message if the private key file cannot be decoded. - Added removal of any excess text from before the "BEGIN" line in the host certificate. 3.32 2013/10/22 (dwd) - Add support to the FilePlugin for reading from http URLs in addition to files on the local disk. If FileBaseDirectory begins with http://, this new feature is enabled. If the http query returns a Content-Length header, the response is streamed through to the client, otherwise the response is read into a memory buffer to determine the length. Last-Modified headers are passed through from the backend http server to the client, and If-Modified-Since is passed through from the client to the backend http server. - Fix bug in the FilePlugin that prevented it from sending the Last-Modified header for files. - Add queuing and keepalive support to the FilePlugin, similar to the SQLPlugin. The maximum number of simultaneous connections are specified in the new config.properties option MaxFileConnections, default 5. The options MaxDbAcquireSeconds and MaxDbExecuteSeconds then become relevant for sending keepalive messages while waiting, even though it isn't technically a "DB". When reading files (as opposed to http urls) the time to "execute" is instantaneous because it just opens the file, so MaxDbExecuteSeconds isn't important then, but when reading http urls the execute time is significant because the http server can take a while to respond. - Improve the accuracy of the active connection count in the log by moving the counter decrement to be before the connection release. - Remove old unused code that supported reading an "xsd_type" from a database table whose name was specified in config.properties option XsdTableName, and that executed a plugin based on the name read from the database. Also remove related config.properties option UseFdoCache. - Remove old unused code related to MonAlisa monitoring, including config.properties options MonitorNode and MonitorMillisDelay. 3.31 2013/10/21 (dwd) - Fix off-by-one error in FilePlugin that had the effect of allowing ".." in the file path. 3.30 2013/06/28 (dwd) - Add support for sending digital signatures on responses instead of an md5 hash, when the client includes the URL parameter "&sec=sig". Also add support for sending the host certificate when the client requests it as type "cert_request". Requires the certificate to be specified in a new parameter 'CertFileName' and the key in a new parameter 'KeyFileName'. - Change to just print one error message to the log when the client drops the connection instead of multiple stack traces. - Change to not print a stack trace when there's an error getting the last-modified time. - Add msecs= on the DB connection acquired message to show how long it took to acquire the connection. - Change internal errors to force a restart of the servlet just like database close problems do. 3.29 2011/05/17 (dwd) - Fixed a bug that corrupts the response when a query for a last-modified time timestamp takes more than 5 seconds, causing a keepalive to be sent. The bug results in the client seeing an error message: XML parse error 7:mismatched tag at line 7 Now if the query takes too long the result is discarded, because it is too late to insert the Last-Modified header after a keepalive is sent. - Add the number of active and remaining connections to the DB connection acquired and released log messages, respectively. 3.28 2011/03/24 (dwd) - Add support for loading locally-accessible files in addition to database queries. Add new FileBaseDirectory property to indicate the base directory of the files that the servlet will load. Make the DataSourceName property no longer required; now at least one of DataSourceName and FileBaseDirectory are required. Corresponding support in the client will be in frontier_client 2.8.1. 3.27 2011/01/20 (dwd) - Change the expiration time on the 503 SERVICE_UNAVAILABLE error to be the time used for errors (the lesser of 5 minutes and the short cache expiration time) instead of the long cache expiration time. 3.26 2011/01/12 (dwd) - Add new SendStaleIfError property, default true, which will send the Cache-control header "stale-if-error" if the client sends in Cache-control: max-stale. This prevents squid from re-using expired data when there's a server error (by default it will reuse data that's up to a week old). Setting max-stale on the client side doesn't affect squid but it indicates that the client is ready to deal with the server errors (frontier_client has been doing that since October 2008 but I never enabled the server side because I thought there was a problem -- details in squid bug #2481). 3.25 2010/12/09 (dwd) - Add new MaxThreads property to be the maximum number of threads that the servlet will allow to wait for a database connection. If a NOT_MODIFIED response can be sent quickly without contacting the database, that will still happen but otherwise if the database needs to be contacted and there are too many threads running the response will be SERVICE_UNAVAILABLE. Default 100. - Ignore exceptions when writing out the transaction end xml. This was causing a MUST NOT HAPPEN HERE exception in cases when the connection had already been dropped by the client and already been reported as an exception. - No longer require the XsdTableName property to be set, as it isn't used by current deployments. 3.24 2010/07/28 (dwd) - Clearly identify the license for the package to be the open source (BSD) Fermitools license. - Support having a slash after the 'Frontier' in the URL in place of the question mark. A slash is better because by default squid does not cache URLs with question marks in them so with slashes a standard squid configuration can be used instead of a custom one. - Add elapsed time to the DB execute log message 3.23 2010/02/12 (dwd) - Fixed race condition in the assignment of thread id numbers. This has been happening for a long time but is now more important because the thread names are used for cleaning up keepalive timer threads. Also include the servlet name in the thread names to ensure no clashes of names between servlets (which has not been seen in practice, it is just in case). 3.22 2009/11/25 (dwd) - There was a serious flaw with the implementation of sending keepalives during DB execute: it was only able to track one at a time, and as a result multiple simultaneous connections to the DB caused a null pointer exception and soon was unable to connect to the DB anymore. It's now able to track any number of simultaneous connections. - Added -Xlint:unchecked javac compile option, and cleaned up all the warnings it added to unchecked generic types. - Made the indentation on keepalive XML elements prettier. 3.21 2009/11/02 (dwd) - Changed the parsing for multiple table names. Now looks for an optional table alias followed by a comma to indicate that another table name is coming, rather than searching for a specific list of keywords. The keywords were matching portions of some table names in CMS. - Added support for continuing to send keepalive messages to the client while waiting for the database to execute a query. This was needed in ATLAS because it has some queries that can take over 10 seconds to execute the first time, until it gets into an Oracle cache. Added config.properties option MaxDbExecuteSeconds to control the amount of time allowed, default 10 seconds. The frontier_client's own readtimeoutsecs option (which also defaults to 10 seconds) is added to that before it times out. - Added an additional config.properties option MaxDbAcquireSeconds which is how long to send keepalives while waiting to acquire the database connection. Defaults to 300 seconds which was the value previously hardcoded. - If a query contains the form "WHERE OWNER=''" (which happened once in CMS) then no longer attempt to look up a modification timestamp for that owner, just let the query proceed without a timestamp. 3.20 2009/10/02 (dwd) - Added a global error to a response where one had inadvertently been left out, after a failure acquiring the database. Global errors cause the client to force a refresh to clean the error out of the cache, and a case was found after a database overload last week where a 'Timed out waiting to acquire the DB' error was cached indefinitely because it wasn't marked as a global error. 3.19 2009/07/14 (dwd) Changes for this release primarily written by David Front. - Support queries on multiple tables at once when tracking modification times. This is needed in order for ATLAS to use Frontier. When multiple tables are in a query, the timestamps on all the tables are looked up in a single database query and the most recent timestamp is cached. - Add new config.properties option 'VerbosityLevel'. When set to 2 or higher, add source file name and line number to each log file line. When set to 4 or higher, add many debug messages. 3.18 2009/05/21 (dwd) - If there's an exception when releaseing the database connection, force the servlet to restart by touching WEB-INF/classes/config.properties. That's quite ugly, but I couldn't think of a better way to do it. We saw a case where this happened and it ended up hanging the servlet and eventually the whole tomcat, so it is important to take of this. NOTE: if installed by root but run under a different user id, as is often the case when it is installed by rpm, this approach will not work! 3.17 2009/02/26 (dwd) - Add new configuration variable LastModifiedTableName that defines the name of the single table where modification times are kept for the whole database, rather than reading them from a LAST_MODIFIED_TIMES table in each account. This is now required if ValidateLastModifiedSeconds is set. 3.16 2008/12/19 (dwd) - Log 'modified at time' also when the modification time was cached - Log 'using cached last-modified time of' the account.table when a cached modification time was found 3.15 2008/12/17 (dwd) - Distinguish in the log when an item has been modified after an if-modified-since by noting it as 'modified at time'. - When the X-Frontier-Id header is missing, still put in the initial log entry as much as is known so we can find out where the request is coming from. 3.14 2008/12/04 (dwd) - Avoid accumulating threads indefinitely when a database is down, by maintaining a lock for getting connections to the database and raising an exception immediately if the 5-minute keepalive thread had already given up by the time the lock is acquired. Previously all the threads would in turn wait 6+ minutes to try to open a socket to the database server and would tie up so many resources that they began to interfere with connections to other databases. - Avoid a couple null pointer exceptions that happened when required pieces were missing from a request. Instead, raise more helpful exceptions for "no query string" and "no X-Frontier-ID". - Also avoid a null pointer exception leading to a "MUST NEVER HAPPEN HERE" internal error when there's no stack trace available in an exception. - Remove an extraneous blank in exception log entries that sometimes made the log appear as if it had an extra blank line. 3.13 2008/11/05 (dwd) - Make sure that a '?' bind variable substitution exists in query before looking for colons, because colon is occasionally used in queries as part of a table value. 3.12 2008/11/04 (dwd) - Add new configuration variable ValidateLastModifiedSeconds which defines the number of seconds between querying for modification times in a LAST_MODIFIED_TIMES table in the same account as each query. The modified time of each table is cached internally for that number of seconds. The time is used to send a Last-Modified header. Squid in turn uses that value once the item expires to send an If-Modified-Since header. When that is received, if the time has not changed send back an http '304 NOT MODIFIED' response. This all combines to allow much more frequent object expirations without overloading the frontier infrastructure, because most times very little data is transferred, only the query and NOT MODIFIED. This requires that every account loaded with this option have a LAST_MODIFIED_TIMES table kept up to date. One way is provided to do this in Oracle, using a PL/SQL script called setup_modtimes.sql. - Use "Cache-Control: max-age" instead of "Expires" headers in the response. Squid handles both and the latter are more difficult to calculate and manage. - Support bind variables in queries, with a question mark where each variable is to be inserted. The values for the variables must be appended to the query, separated by colons (:). This helps performance a bit at least on Oracle, because it is able to cache the parsed value of the query and re-use it rather than caching many similar queries. - Handle global errors more robustly, so they should always be able to be parsed by the client. Previously some kinds of errors would result in invalid xml code in the response. - Give more debug information in the log if the command parser sees a premature end of command. 3.11 2008/04/14 (dwd) - Set the maximum keepalive time (which includes the time it takes to allocate a connection to the DB) to 5 minutes 3.10 2008/02/28 (dwd) - Set the expiration time for exception messages to 5 minutes. 3.9 2007/10/24 (dwd) - When a blob is null, send a null indication rather than empty. This is the more correct way to handle null blobs than what was put in in release 3.5. 3.8 2007/05/24 (dwd) - Back out change from 3.7 because that ends up eliminating the Content-Length header from even small responses, causing squid to drop the connection after every query instead of keeping persistent connections. Instead, we will rely on squid 2.6's "collapsed_forwarding" feature to maximize sharing. Also fix an inadvertent too-early sending of the response header that was put in in version 3.5. Add a log message about the response being precommitted whenever the header is committed early, resulting in the removal of Content-Length. This now happens only when responses take up more than the default 8K buffer in the ServletOutputStream. 3.7 2007/05/01 (dwd) - Send response headers as soon as possible to maximize the sharing that squid does. 3.6 2007/04/05 (dwd) - Get the transaction id (with a -ka suffix) into log messages from the keepalive thread 3.5 2007/03/22 (dwd) - Cleaned up log messages so those that occur regularly all will show timestamp and thread id - Greatly improved performance for blobs, for small rows, and for zipping - Replaced Base64 encode/decode implementation with much simpler/ cleaner version - Eliminate null pointer exception on null blobs - Renamed CacheExpireSeconds and CacheExpireHourOfDay to LongCacheExpireSeconds and LongExpireHourOfDay. Added two new parameters ShortCacheExpireSeconds and ShortCacheExpireHourOfDay which are the same except apply to all queries that have "&ttl=short" (ttl stands for time-to-live). The default on ShortCacheExpireSeconds is 0. - Catch all Throwables, not just Exceptions, so that Errors are also included and debugged - When an Exception or Error occurs pass the function, source file number, and line number to the application in addition to the name of the throwable - The "stop" log message now includes the entire transaction, so the number of active threads and the elapsed time are more meaningful 3.4 2006/10/26 (dwd) - Added support for oracle BLOB (Binary Large OBject) types - Added two optional parameters to WEB-INF/classes/config.properties: CacheExpireSeconds - number of seconds after queries in which the caching of the query should expire. Default is 7 days of seconds, 604800. CacheExpireHourOfDay - the hour of the day at which all query caching should expire, if less than CacheExpireSeconds. For example, if it is set to "23", all items will expire at 11:00 p.m. in the local time of the server. Default is to use CacheExpireSeconds instead. 3.3 2006/09/05 (dwd) - Fixed bug where the keepalive message task wasn't shut down if the database connection failed - Added a refinement to Luis Ramos' patch of 3.1: don't add the parenthesized precision after the NUMBER data type if the precision is zero 3.2 2006/07/26 (dwd) - Added sending a keepalive message to the client every 5 seconds while waiting to acquire a database connection, to avoid client timeouts under heavy load - Added sending the servlet version to the catalina log and the client - Reduced non-useful debugging output to the catalina log and improved some others 3.1 2006/06/07 (dwd) - Added feature to zip the retrieved payload using gzip levels 0-9 where level 0 is no zipping, level 1 is fastest but least compressed, and level 9 is slowest but most compressed - Added precision in parentheses to the NUMBER data type. Requested by Luis Ramos of CERN and patch supplied by him - Cleaned up some exception message typos 3.0 and earlier releases did not have release notes