Dynamic Fields

DSE Version: 6.0



Hi, I'm Joe Chu and in this video we'll be talking about dynamic fields.

If you remember anything about dynamic fields in this video, then hopefully it will be this very first point that I'll make, which is that you should avoid using these whenever possible. While dynamic fields is a pretty cool feature, which you'll see more of here, it is something that can be hard to control, particularly in the way that it consumes memory.

The problem is, the more memory that is being allocated to be used by a dynamic field, the less free memory this is, which can lead to issues with performance or errors as a results of not having enough memory available.

Since the Cassandra map data type is tied to using dynamic field for search, this means that each key value that you add to a column is going to automatically consume more and more memory. If you do not have limits on the number of keys in these map columns, then you too may soon experience these issues.

Ok, if you're still with me at this point, then apparently I was not able to scare you off. In that case let's talk about what a dynamic field is, and how they work.  

A dynamic field is a special type of field that allows additional fields to be indexed without being explicitly defined in the search index schema. As long as the name of the field matches a certain pattern that is set by the dynamic field, then they'll also be indexed.

As an example, let's say that you define a dynamic field with the name user underscore star. If there are other fields that are created, like user underscore age, user underscore gender, user underscore height, and so forth, then those fields will automatically be indexed, and searchable.  This maybe have some use when the data you receive for indexing is not consistent, or does not follow a strict schema and allows for arbitrary field names.

As mentioned in the beginning, one of the reasons why we do talk about dynamic fields though, is because the Cassandra map data type is automatically defined as a dynamic field in the search index schema. The dynamic field is ideally suited for a map since keys can be arbitrary, with no requirement for the number of keys, the inclusion of certain keys, or whether each row contains the same set of keys in the map column.

Each key technically becomes a field, and the key value is automatically indexed thanks to the dynamic field.

There are some requirements though, in order for map columns to work in Search. Since a field name is suppose to be a string, that means only maps with the key datatype set to text can be used. The key value does not have this limitation though, and can use data types other than text. Finally the key needs to include a prefix or suffix, which should somewhat match the column name, i.e. the dynamic field name.

Let's take a look at an example. Here we have a users table with a map column called phonenumber underscore, and uses the text datatype for both the key and key value. The table shows several rows and key-value pairs found in the phone number underscore column. Note that key names also start with phone number underscore, since that is the pattern that will allow the dynamic field to automatically index those keys as new fields.

For the search index schema definition, the list of fields should contain  a dynamicField element, which shows the name to be phonenumber underscore *, which is derived from the column name and is what is used to match new fields to be indexed. The type is a strfield, which corresponds to the text datatype used for the key value.

Here there are two possible search queries that we can execute for the map column. One is to find all occurrences of a particular key, which shown here is phone number underscore home.  Another way would be to search for a specific value in a key, with this example looking for rows with the work phone number 410-715-6911. Unfortunately, it is not possible to value across all keys.

No write up.
No Exercises.
No FAQs.
No resources.
Comments are closed.