Otdfbt

mod_rewrite syntax order

mod_rewrite has some specific ordering rules that affect processing. Before anything gets done, the RewriteEngine On directive needs to be given as this turns on mod_rewrite processing. This should be before any other rewrite directives.

RewriteCond preceding RewriteRule makes that ONE rule subject to the conditional. Any following RewriteRules will be processed as if they were not subject to conditionals.

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule $/blog/(.*).html $/blog/$1.sf.html

In this simple case, if the HTTP referrer is from serverfault.com, redirect blog requests to special serverfault pages (we're just that special). However, if the above block had an extra RewriteRule line:

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule $/blog/(.*).html $/blog/$1.sf.html
RewriteRule $/blog/(.*).jpg $/blog/$1.sf.jpg

All .jpg files would go to the special serverfault pages, not just the ones with a referrer indicating it came from here. This is clearly not the intent of the how these rules are written. It could be done with multiple RewriteCond rules:

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule ^/blog/(.*).html /blog/$1.sf.html
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule ^/blog/(.*).jpg /blog/$1.sf.jpg

But probably should be done with some trickier replacement syntax.

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule ^/blog/(.*).(html|jpg) /blog/$1.sf.$2

The more complex RewriteRule contains the conditionals for processing. The last parenthetical, (html|jpg) tells RewriteRule to match for either html or jpg, and to represent the matched string as $2 in the rewritten string. This is logically identical to the previous block, with two RewriteCond/RewriteRule pairs, it just does it on two lines instead of four.

Multiple RewriteCond lines are implicitly ANDed, and can be explicitly ORed. To handle referrers from both ServerFault and Super User (explicit OR):

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$) [OR]
RewriteCond %HTTP_REFERER ^https?://superuser.com(/|$)
RewriteRule ^/blog/(.*).(html|jpg) /blog/$1.sf.$2

To serve ServerFault referred pages with Chrome browsers (implicit AND):

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteCond %HTTP_USER_AGENT ^Mozilla.*Chrome.*$
RewriteRule ^/blog/(.*).(html|jpg) /blog/$1.sf.$2

RewriteBase is also order specific as it specifies how following RewriteRule directives handle their processing. It is very useful in .htaccess files. If used, it should be the first directive under "RewriteEngine on" in an .htaccess file. Take this example:

RewriteEngine On
RewriteBase /blog
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule ^(.*).(html|jpg) $1.sf.$2

This is telling mod_rewrite that this particular URL it is currently handling was arrived by way of http://example.com/blog/ instead of the physical directory path (/home/$Username/public_html/blog) and to treat it accordingly. Because of this, the RewriteRule considers it's string-start to be after the "/blog" in the URL. Here is the same thing written two different ways. One with RewriteBase, the other without:

RewriteEngine On

##Example 1: No RewriteBase##
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule /home/assdr/public_html/blog/(.*).(html|jpg) $1.sf.$2

##Example 2: With RewriteBase##
RewriteBase /blog
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule ^(.*).(html|jpg) $1.sf.$2

As you can see, RewriteBase allows rewrite rules to leverage the web-site path to content rather than the web-server, which can make them more intelligible to those who edit such files. Also, they can make the directives shorter, which has an aesthetic appeal.

RewriteRule matching syntax

RewriteRule itself has a complex syntax for matching strings. I'll cover the flags (things like [PT]) in another section. Because Sysadmins learn by example more often than by reading a man-page I'll give examples and explain what they do.

RewriteRule ^/blog/(.*)$ /newblog/$1

The .* construct matches any single character (.) zero or more times (*). Enclosing it in parenthesis tells it to provide the string that was matched as the $1 variable.

RewriteRule ^/blog/.*/(.*)$ /newblog/$1

In this case, the first .* was NOT enclosed in parens so isn't provided to the rewritten string. This rule removes a directory level on the new blog-site. (/blog/2009/sample.html becomes /newblog/sample.html).

RewriteRule ^/blog/(2008|2009)/(.*)$ /newblog/$2

In this case, the first parenthesis expression sets up a matching group. This becomes $1, which is not needed and therefore not used in the rewritten string.

RewriteRule ^/blog/(2008|2009)/(.*)$ /newblog/$1/$2

In this case, we use $1 in the rewritten string.

RewriteRule ^/blog/(20[0-9][0-9])/(.*)$ /newblog/$1/$2

This rule uses a special bracket syntax that specifies a character range. [0-9] matches the numerals 0 through 9. This specific rule will handle years from 2000 to 2099.

RewriteRule ^/blog/(20[0-9]2)/(.*)$ /newblog/$1/$2

This does the same thing as the previous rule, but the 2 portion tells it to match the previous character (a bracket expression in this case) two times.

RewriteRule ^/blog/([0-9]4)/([a-z]*).html /newblog/$1/$2.shtml

This case will match any lower-case letter in the second matching expression, and do so for as many characters as it can. The . construct tells it to treat the period as an actual period, not the special character it is in previous examples. It will break if the file-name has dashes in it, though.

RewriteRule ^/blog/([0-9]4)/([-a-z]*).html /newblog/$1/$2.shtml

This traps file-names with dashes in them. However, as - is a special character in bracket expressions, it has to be the first character in the expression.

RewriteRule ^/blog/([0-9]4)/([-0-9a-zA-Z]*).html /newblog/$1/$2.shtml

This version traps any file name with letters, numbers or the - character in the file-name. This is how you specify multiple character sets in a bracket expression.

RewriteRule flags

The flags on rewrite rules have a host of special meanings and usecases.

RewriteRule ^/blog/([0-9]4)/([-a-z]*).html /newblog/$1/$2.shtml [L]

The flag is the [L] at the end of the above expression. Multiple flags can be used, separated by a comma. The linked documentation describes each one, but here they are anyway:

L = Last. Stop processing RewriteRules once this one matches. Order counts!
C = Chain. Continue processing the next RewriteRule. If this rule doesn't match, then the next rule won't be executed. More on this later.
E = Set environmental variable. Apache has various environmental variables that can affect web-server behavior.
F = Forbidden. Returns a 403-Forbidden error if this rule matches.
G = Gone. Returns a 410-Gone error if this rule matches.
H = Handler. Forces the request to be handled as if it were the specified MIME-type.
N = Next. Forces the rule to start over again and re-match. BE CAREFUL! Loops can result.
NC = No case. Allows jpg to match both jpg and JPG.
NE = No escape. Prevents the rewriting of special characters (. ? # & etc) into their hex-code equivalents.
NS = No subrequests. If you're using server-side-includes, this will prevent matches to the included files.
P = Proxy. Forces the rule to be handled by mod_proxy. Transparently provide content from other servers, because your web-server fetches it and re-serves it. This is a dangerous flag, as a poorly written one will turn your web-server into an open-proxy and That is Bad.
PT = Pass Through. Take into account Alias statements in RewriteRule matching.
QSA = QSAppend. When the original string contains a query (http://example.com/thing?asp=foo) append the original query string to the rewritten string. Normally it would be discarded. Important for dynamic content.
R = Redirect. Provide an HTTP redirect to the specified URL. Can also provide exact redirect code [R=303]. Very similar to RedirectMatch, which is faster and should be used when possible.
S = Skip. Skip this rule.
T = Type. Specify the mime-type of the returned content. Very similar to the AddType directive.

You know how I said that RewriteCond applies to one and only one rule? Well, you can get around that by chaining.

RewriteEngine On
RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$)
RewriteRule ^/blog/(.*).html /blog/$1.sf.html [C]
RewriteRule ^/blog/(.*).jpg /blog/$1.sf.jpg

Because the first RewriteRule has the Chain flag, the second rewrite-rule will execute when the first does, which is when the previous RewriteCond rule is matched. Handy if Apache regular-expressions make your brain hurt. However, the all-in-one-line method I point to in the first section is faster from an optimization point of view.

RewriteRule ^/blog/([0-9]4)/([-0-9a-zA-Z]*).html /newblog/$1/$2.shtml

This can be made simpler through flags:

RewriteRule ^/blog/([0-9]4)/([-0-9a-z]*).html /newblog/$1/$2.shtml [NC]

Also, some flags also apply to RewriteCond. Notably, NoCase.

RewriteCond %HTTP_REFERER ^https?://serverfault.com(/|$) [NC]

Will match "ServerFault.com"

What is the fundamental format and
structure of mod_rewrite rules?

I'll defer to sysadmin1138's excellent answer on these points.

What form/flavor of regular
expressions do I need to have a solid
grasp of?

In addition to the syntax order, syntax matching/regular expressions, and RewriteRule flags outlined by sysadmin1138, I believe it bears mentioning that mod_rewrite exposes Apache environment variables based on HTTP request headers and Apache's configuration.

I would recommend AskApache's mod_rewrite Debug Tutorial for a comprehensive list of variables which may be available to mod_rewrite.

What are the most common
mistakes/pitfalls when writing rewrite
rules?

Most problems with RewriteRule's stem from a misunderstanding of PCRE syntax/failure to properly escape special characters or a lack of insight into the content of the variable(s) used for matching.

Typical problems and recommended troubleshooting:

• 
  500 - Internal Server Error - Remove Windows carriage controls in configuration file(s) if present, make sure mod_rewrite is enabled (wrap directives in IfModule conditional to avoid this scenario), check directive syntax, comment out directives until problem is identified


• 
  Redirect loop - Make use of RewriteLog and RewriteLogLevel, comment out directives until problem is identified

What is a good method for testing and
verifying mod_rewrite rules?

First, look at the contents of the environment variable(s) you plan to match against - if you have PHP installed, this is as simple as adding the following block to your application:

<?php
 var_dump($_SERVER);
?>

... then write your rules (preferably for testing on a development server) and note any inconsistent matching or activity in your Apache ErrorLog file.

For more complex rules, use mod_rewrite's RewriteLog directive to log activity to a file and set RewriteLogLevel 3

Are there SEO or performance
implications of mod_rewrite rules I
should be aware of?

AllowOverride all impacts server performance as Apache must check for .htaccess files and parse directives with each request - if possible, keep all directives in the VirtualHost configuration for your site or enable .htaccess overrides only for the directories which need them.

Google's Webmaster Guidelines explicitly state: "Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as 'cloaking.'" - avoid creating mod_rewrite directives which filter for search engine robots.

Search engine robots prefer a 1:1 content:URI mapping (this is the basis for ranking links to content) - if you are using mod_rewrite to create temporary redirects or you are serving the same content under multiple URI's, consider specifying a canonical URI within your HTML documents.

Are there common situations where
mod_rewrite might seem like the right
tool for the job but isn't?

This is a huge (and potentially contentious) topic in its own right - better (IMHO) to address uses on a case-by-case basis and let askers determine whether the resolutions suggested are appropriate to their needs.

What are some common examples?

AskApache's mod_rewrite Tricks and Tips covers just about every common use-case that pops up regularly, however, the "correct" solution for a given user may depend upon the sophistication of the user's configuration and existing directives (which is why it is a generally a good idea to see which other directives a user has in place whenever a mod_rewrite question comes up).

Like many admin/developers I've been fighting the intricacies of rewrite rules for years and am unhappy with the existing Apache documentation, so I decided as a personal project to get to the bottom of how mod_rewrite actually works and interacts with the rest of the Apache core, so over the last few months I've been instrumenting test cases with strace + drilling into the source code to get a handle on all of this.

Here are some key comments that rewrite rule developers need to consider:

• Some aspects of rewriting are common to server config, virtual host, directory, .htaccess processing however
  


• Some processing is very different for the root config (server config, virtual host and directory) as opposed to the PerDir (.htaccess) processing.


• Worse because PerDir processing can almost indiscriminately trigger INTERNAL REDIRECT cycling, the root config elements have to be written aware that such PerDir processing can trigger this.

I would go as fas as to say that because of this you almost need to split the rewrite user communities into two categories and treat them as entirely separate:

• Those with root access to the Apache config. These are typically admin/developer with an application dedicated server/VM, and the message here is quite simple: avoid using .htaccess files if at all possible; do everything in your server or vhost config. Debugging is reasonable easy since the developer can set debugging and has access to the rewrite.log files.



• 
  

  Users of a shared hosted service (SHS).

  
  
  
  • Such users have to use .htaccess / Perdir processing as there is no alternative available.
  
  
  • Worse, the skill level of such users (as far as using the regexp driven ladder-logic of mod_rewrite) is generally significantly less than experienced admins.
  
  
  • Apache and the hosting providers offer no debugging / diagnostic support. The only diagnostic information is a successful redirection, a redirection to the wrong URI. or a 404/500 status code. This leaves them confused and helpless.
  
  
  • Apache is extremely weak explaining how rewriting works for this use case. For example it does not provide a clear explanation of what PerDir .htaccess file is selected and why. It does not explain the intricacies of PerDir cycling and how to avoid this.
  
  

There is possibly a third community: admin and support staff in SHS providers who end up with a foot in both camps and have to suffer the consequences of the above.

I have written a couple of article-style blog posts (e.g More on using Rewrite rules in .htaccess files) which covers a lot of detailed points which I won't repeat here to keep this post short. I have my own shared service as well as supporting some dedicated & VM FLOSS projects. I started out using a standard LAMP VM as a test vehicle for my SHS account, but in the end I found it better to do a proper mirror VM (described here).

However, in terms of how the admin community should support .htaccess users, I feel that we need to develop and to offer:

• A coherent description of how the rewrite system actually works in PerDir processing


• A set of guidelines/best practices on how to write .htaccess rewrite rules


• A simple web based rewrite script parser sort of similar to the W3C html parsers, but by which users can input test URIs or test vectors of the same and get an immediate log of the rewrite logic flow/


• 
  

  Hints on how to get built-in diagnostics from your rules (e.g.

  
  
  
  • Use [E=VAR:EXPR] exploiting the fact that EXPR will expand backreferences ($N or %N) to make them available as diagnostics to the target script.
  
  
  • 
    

    If you topically order your rewrite rules using [OR],[C],[SKIP] and [L] flags so that the entire rewrite scheme works without the need to exploit internal redirection, then you can add the following as rule 1 to avoid all looping hassle:

    
    
    

    RewriteCond %ENV:REDIRECT_STATUS !=""
    RewriteRule . - [L]
    

    
    
  
  

Using rewritemap

There are lots of things you can do with rewritemaps. Rewritemaps get declared using the Rewritemap directive, and can then be used both in RewritCond evaluations, and in RewriteRule Subsitutions.

The general syntax for RewriteMap is:

RewriteMap MapName MapType:MapSource

For example:

RewriteMap examplemap txt:/path/to/file/map.txt

You can then use the mapname for constructs like this:

$examplemap:key

The map contains key/value pairs. If the key is found, the value is subsituted. Simple maps are just plain text files, but you can use hash maps, and even SQL queries. More details are in the docs:

http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewritemap

Unescaping strings.

There are four internal maps you can use to do some manipulations. Especially unescaping strings can come in handy.

For example: I want to test for the string "café" in the query string. However, the browser will escape this before sending it to my server, so I 'll need to either figure out what the URL escaped version is for every string I wish to match, or I can just unescape it...

RewriteMap unescape int:unescape

RewriteCond %QUERY_STRING (location|place)=(.*)
RewriteCond $unescape:%2 café
RewriteRule ^/find/$ /find/1234? [L,R]

Note how I use one RewriteCond to just capture the argument toe the query string parameter, and then use the map in the second rewriteCond to unescape it. This then gets compared.
Also note how I need to us %2 as key in the rewritemap, as %1 will contain either "location" or "place". When you use parentheses to group patterns they will also be captured, wether you plan to use the result of the capture or not...

What are the most common
mistakes/pitfalls when writing rewrite
rules?

A really easy pitfall is when you rewrite URLs that alter the apparent path, e.g. from
/base/1234/index.html to /base/script.php?id=1234. Any images or CSS with relative paths to the script location will not be found by the client. A number of options to resolve this can be found on this faq.