Searching base64 encoded text

One rare aspect of POG that may require some ‘adaptation’ is that, by default, POG encodes all strings to base64 before inserting them in the database. This is done so that all dynamic SQL queries created by POG are SQL-safe and takes the hassle of having to escape unsafe strings away from the PHP developer. Another reason for base64 is that it is a database-independent format, and makes any pog-related code more portable, since POG supports multiple databases (mysql, postgresql, sqlite and so on).

Read more about base64 and POG

The one drawback that had to be worked around is that it is difficult to perform partial string matching search on a base64 encoded text. This is simply because of the way base64 encodes text. For example, the string

The quick fox jumped over the lazy dog

has a base64 encoded value of

VGhlIHF1aWNrIGZveCBqdW1wZWQgb3ZlciB0aGUgbGF6eSBkb2cg

So, if I were searching a database for this exact string, it wouldn’t be hard at all since I could create a query that looks like this:

select record where base64_encoded_text = “VGhlIHF1aWNrIGZveCBqdW1wZWQgb3ZlciB0aGUgbGF6eSBkb2cg”;

However, if I’m searching for all records that contain the word “dog” (“ZG9n” in base64), I can’t create the following query:

*select record where base64_encoded_text like ’% ZG9n %’*

since the string ZG9n is not present in VGhlIHF1aWNrIGZveCBqdW1wZWQgb3ZlciB0aGUgbGF6eSBkb2cg even though they both contain the word dog when decoded.

2 solutions:
There are 2 workarounds for this particular issue that I’ve identified up to now, and you can find them below. Since this is one of the most frequently asked question regarding POG, I hope these will be useful to you:

Solution 1: Get all records, then filter
In this scenario, the developer programs ‘around’ POG’s limitation and gets all records, or a subset of the records from the database using POG’s GetList() CRUD method. Then, the developer loops through all the returned records (which are POG objects), and then filters by the attribute being searched.

Example:

function Search($needle)
{
$filteredList = array();
$bookList = $book -> GetList (array(array(“bookId”, ”>”, 0)));
foreach ($bookList as $aBook)
{
if (strpos($aBook->text, $needle)
{
$filteredList[] = $aBook;
}
}

return $filteredList();
}

The drawback in this solution is that the developer first needs to get all potential records and essentially do the work of the database. The key word here is potential. It is strongly suggested to find another condition that will reduce the number of potential records that need to be filtered.

Solution 2: Implement base64 decode in the database
This is a solution that can be extremely useful in big projects. I’ve personally used it in a large project I needed to tackle while using POG. The rational behind this method is as follows:

If we teach the database to decode base64 strings, then we don’t have to return all potential records and filter them. Instead, we can let all the processing to the database, making it faster and simpler to code on the front-end. If the database has a function base64_decode, we can simply issue the following query:

*select base64_decode(record) where plain_text like ’% dog %’;*

Thus, instead of encoding the partial string we’re looking for, we simply decode the entire string on the database, and match the result with our partial string. This ensures that we’ll get all records that contain the word dog.

Howto:
MySQL doesn’t yet support base64 decode, so you’ll need to create a custom function to do that. As a starting point, feel free to take a look at this custom function base64.sql provided by Ian Gulliver (http://www.firestuff.org)

Once your database is able to decode base64 strings, simply implement a Search() Method in your POG object(s) which queries the database using base64_decode.

Previous discussions on partial string matching:
Google group discussion #1
Google group discussion #2

Final Note: In POG 2.0+, developers are able to disable automatic encoding/decoding from within the POG’s configuration file, thus side-stepping the task of finding an elegant solution for partial string matching in base64-encoded strings.