NI DIAdem (How I Learned To Stop Worrying And Love Big Data)

Model based software design.
Model based software design.

If you have lots of data or find yourself doing repetitive tasks with small sets of data, then you might want to consider NI DIAdem software to navigate, view, analyze, and report your data.
Data Mining
DIAdem works with NI Datafinder to index your data enabling powerful data mining tools. NI Datafinder works with several file formats: Excel, CSV, TDM, LVM and many others. If it lacks native support for your data file, try downloading a plugin for NI DIAdem from the links at the bottom of this post.
Once NI Datafinder has indexed your data, you can perform saveable and scriptable queries on the data in the Navigation pane. You can tailor your queries to include file names, file dates, authors, and you can also search within the data values themselves. You can, for example, search for all data from Station 1 collected June 1, 2011 through August 30, 2011 with an average temperature of 30C or higher. You can further extend the query capabilities and convenience using scripting.
Viewing Data
Use the DIAdem viewing pane to easily inspect and visualize data. Once you have your data loaded into the data portal using the Navigation pane, you can plot it by dragging and dropping into plots you have configured in the viewing pane and perform calculations. You can also use the viewing pane to “link” data from several sources collected of the same event and synchronize playback. Scripting common plotting tasks is also possible.
Reporting Data
DIAdem report pane builds customized publication quality data plots. In this pane you have more control over how and where the data appears. You can add icons and logos for your organization. Also, you can use variables to call out important statistics (for example) in the graphs on the report. Scripting also supports updating these variables allowing scripts to query and update the report template for the data. Reports can be exported into PDF, PPT, and HTML formats. Learn more about report generation options in the DIAdem REPORT image gallery.

In my opinion, DIAdem is a powerful tool for mining, inspecting, and reporting big data sets. Further, productivity and consistency can be enhanced using scripts.

Further Reading
NI DIAdem
NI Datafinder
Scripting with NI DIAdem
Data Mining Tools
DIAdem REPORT image gallery
Downloading the DIAdem Plugin