Datatable

From Emergent

A Datatable is a tabular data structure that holds data in rows and columns. Each table has a fixed number of columns; each column holds items of a specific data type. The table can have 0 or more rows of data. New data can be appended at the end, or inserted anywhere in the table. Any row or rows can be deleted. Datatables can be saved in files, and imported and exported.

The Datatable plays a key role in Emergent. If you learn to think in terms of datatables, and operations on them, you will find that many things are much easier to do.

Contents 1 Columns 1.1 Scalar Column 1.2 Matrix Column 2 Tabular data types 2.1 Int 2.2 Float 2.3 Double 2.4 String 2.5 Byte 2.6 Variant (any type) 3 Creating and Modifying Datatables 3.1 To create a datatable 3.2 To add a column 3.3 To delete a column 3.4 To change the data type of a column 3.5 To change a column from scalar to matrix, or change the matrix geometry 3.6 To move a column 4 Adding and changing data 4.1 Edit data using the data editor 4.1.1 To add (or insert) rows 4.1.2 To edit cells 4.1.3 To delete rows 4.2 Copy and paste data 5 Specialized Data Processing Operations 6 Local Read and Write Index Iterator Values 7 Accessing Matrix Data within DataTables 8 Reference Information 9 Multi-dimensional loading format

Columns

Each column is of a fixed type (see below); the total number of rows is the same for all columns. A row-column location is called a cell. Each cell can be either a single value (a scalar column), or multiple values (a matrix column).

Scalar Column

The most common type of column is a scalar column; in this column, each cell holds a single value. A datatable of all scalar columns would be similar to a spreadsheet or a database table.

Matrix Column

A matrix column has cells that are a matrix of some basic type. Each cell in the column has exactly the same dimensions.

Matrix columns are especially useful for representing input or output patterns of network layers; in this case, the cell typically has a 2d geometry that is the same as that of the layer. (If you are using layer groups, then you can use either 2d or 4d cells.)

Tabular data types

All the values in a column are of a single type. Below are listed the available data types for columns.

Int

An int is a 32-bit integral value that can store numbers from −2,147,483,648 to +2,147,483,647.

Int is a good choice when dealing with counts, or indexes.

Float

A float is a (32-bit) floating point number that can represent about 7 significant digits of precision, with a magnitude between about 10^-38 to 10^38.

Float is a good choice when dealing with any kind of calculated or statistical value, or real-valued parameters and values.

Double

A double is a (64-bit) floating point number that can represent about 15 significant digits of precision, with a magnitude between about 10^-308 to 10^308.

Double may be a good choice when dealing with numbers that require extended accuracy, or whose values could be extremely large or small. Note that computational operations on doubles are significantly slower than on floats.

String

A String can hold a textual value (in ANSI encoding) of any practical length (up to available memory.)

Byte

A Byte holds a small unsigned 8-bit number from 0 to 255. (This type is not often used, and is primarily provided to efficiently store large image arrays.)

Variant (any type)

A Variant is a flexible type that can store a value of any other type. Each Variant keeps track of the type of the value, and the value itself. Although Variants are flexible, they take more memory than the specific types. In general, you should use a specific type unless you have a reason to require a Variant.

Creating and Modifying Datatables

To create a datatable

1. select the data group in which you want to create the table
2. right click, and select New
3. click Ok
4. select the new table, and create columns

To add a column

1. select the datatable
2. right click, and select Columns/NewCol (or NewColMatrix for matrix col)
3. choose the data type for the column
4. enter a name (see Object Naming for rules)

To delete a column

1. select the column in the tree view or the column panel
2. right click, and select Delete

To change the data type of a column

1. select the column in the tree view or the column panel
2. right click, and select Columns/Change Col Type
3. select the new data type

Emergent will preserve data where it can, such as when changing from float to double.

To change a column from scalar to matrix, or change the matrix geometry

1. select the column in the tree view or the column panel
2. right click, and select Columns/Change Col Cell Geom
3. enter the new geometry

Emergent will preserve data where it can; but if you add or remove whole dimensions, the resulting position of the data is undefined. Note that you cannot change both the type and matrix geometry in one step.

To move a column

You can change the order of columns. The easiest way is by drag and drop

1. select the column you wish to move
2. hold down the mouse button and drag the column to the new position you would like (if you want it at the end, drag it onto the table itself)
3. release the mouse; click Move Here (or if moving to end, when on table, click Move Into)

Adding and changing data

Emergent provides many ways to add and modify data in Datatables. You can:

• edit the data manually, using the table editor
• import or paste data from an external source, such as a spreadsheet
• use a DataProc operation
• write a Program to generate the data

Edit data using the data editor

The data editor in the Edit Panel lets you:

• append, insert, or delete a row or rows
• edit the data value in cells, or in matrix cells
• copy or paste the data to/from the clipboard

To add (or insert) rows

1. click on an existing row or rows (the left-hand buttons beside each row of data) - the number of rows you select will be the number of rows added (inserted)
2. right click, and select Append Rows (or Insert Rows)

The rows will be appended (or inserted at the selection point).

You can also add rows from the table context menu or edit panel menu: Rows/Add (or Insert) Rows.

To edit cells

1. click in the cell whose value you wish to change - for matrix cells, you must first click the matrix cell, then you can use the matrix cell editor at the bottom of the panel to edit the matrix cell value you want
2. type the new value
3. click or tab to a new cell, to save your changes to the table

To delete rows

1. select the row or rows you wish to delete
2. right click, and select Delete Rows

The rows will be deleted.

You can also delete rows from the table context menu or edit panel menu: Rows/Delete Rows.

Copy and paste data

You can copy and paste data between Emergent datatables, Emergent matrices, and many external programs that support tabular data, such as spreadsheets (Excel, OpenOffice Calc, etc.).

1. select the block of data you wish to copy in the source datatable, program, or matrix
2. in the destination datatable, click on the upper left cell of the area in which you want to paste
3. right click, and select Paste

If you copy more cells than can fit in the area you select, the remainder will be ignored.

When you copy data from a range of cells that includes matrix columns, Emergent treats each row as having a number of subrows equal to the largest matrix cell; when copying from a datatable, "missing" values in smaller cells are blank; when copying to a datatable, data corresponding to these "missing" locations is ignored. To avoid confusion, you may wish to operate on such data one column at a time.

Specialized Data Processing Operations

There are several collections of data processing operations that can be performed on data tables -- see DataProc for details.

You can also write a small Program to generate or modify data. You can access the operations available interactively via the data_proc collection in your program. In addition, the GSL (Gnu Scientific Library) is available, as well as many built-in statistical and analytical operations.

Local Read and Write Index Iterator Values

DataTables contain their own read and write index values, which can be set by ReadItem and WriteItem methods, and accessed by ReadIndex and WriteIndex. This allows a data table to have its own "current row" value that persists across different Programs, for example. This is used in the application of InputData to a Network, for example.

Accessing Matrix Data within DataTables

• See Matrix from DataTable for information on how to access/modify Matrix information within DataTable objects.

Reference Information

The following is automatically generated from the source code:

DataTable

Multi-dimensional loading format

If you want to import multidimensional data into emergent, the fastest way to determine the format is to create a DataTable to your specifications, save it to a file, and then inspect the file. Here is an example of 2x2 DataTable of type Matrix, where each of the four matrices are also 2x2 and contain Ints. The emergent object file for the datatable used in this example is available as Test.dtbl. You can load it by going to Object > Load. After loading, if you right click on the context menu for this DataTable, you'll see an option to Save Data in a TAB delimited format. The format for the DataTable just described looks like so:

H:     &test[2:0,0]<2:2,2>     &test[2:1,0]    &test[2:0,1]    &test[2:1,1]    &test2[2:0,0]<2:2,2>    &test2[2:1,0]   &test2[2:0,1]   &test2[2:1,1]
_D:     1       2       3       4       9       10      11      12
_D:     5       6       7       8       13      14      15      16

H: stands for header, and _D: stands for data. &test is a reference to the name of the datatable, test.